Game-Theoretic Robust Reinforcement Learning Handles Temporally-Coupled Perturbations

We thank Reviewer GZCq for the valuable feedback on our paper. We have provided detailed responses to the concerns raised by the reviewer and updated our paper, incorporating clarifications and discussions where necessary.

1. This paper does not have a clear mathematical representation of the problem it intends to address.

We appreciate the reviewer's feedback regarding the clarity of our problem formulation. In response to this concern, we have enhanced the paper by providing a more explicit and detailed formulation of our problem in Section 2 based on the zero-sum game. We thank the reviewer for highlighting the mathematical problem formulation.

2. The article claims its primary contribution lies in using zero-sum games to formulate the robust RL problem. However, employing zero-sum games to account for uncertainties, whether in single-agent or multi-agent RL, is well-established. Given this widespread application, the paper's stated novelty becomes questionable.

While we acknowledge the precedent exploration of zero-sum games in robust RL within existing works, it is crucial to emphasize the distinctive contribution of our method, GRAD.

• Converge to approximate equilibriums on an adversary population.

During training, our approach achieves an approximate equilibrium on an adversary policy set, which is totally different from other robust RL papers which use the zero-sum game formulation but only learn with an adversary during alternating training. GRAD has the capability to continuously explore and learn new policies as best responses that are not present in the current policy set, which allows for a more thorough exploration of the policy space.

• Adaptable to different adversaries.

Significantly, prior works, such as [1, 2, 3], primarily focus on a specific adversary or only consider the worst-case scenario, often adopting a highly pessimistic outlook that limits their adaptability to adversaries with different constraints. However, GRAD does not specifically target certain adversaries during training. Instead, at each epoch, it learns the best response to the adversary policy sampled from the adversary population, making it adaptable to different adversarial scenarios, including both temporally-coupled and non-temporally-coupled adversaries.

Furthermore, previous approaches exclusively consider specific attack domains for adversaries, such as perturbations on environmental dynamics [1, 2] and action perturbations [3]. In contrast, GRAD represents a pioneering effort by introducing the first robust RL framework that accommodates various adversaries including action, state, and transition adversaries, without any constraints on attack domains. This broader scope positions GRAD as a more practical and scalable general solution for robust RL.

• Solving adversarial RL rather than uncertainty-aware RL.

Our method is proposed to defend against adversarial perturbations and improve adversarial robustness, while many papers such as [4] only focus on uncertainty-aware RL. Though GRAD exhibits strong performance even under model uncertainty, as shown in our experiments. Adversarial RL where the learning process is intentionally disrupted by an adaptive adversary (as a maxmin problem) is more challenging than uncertainty-aware RL (a max problem under uncertainty).

To provide further clarity on this aspect, we have included a detailed illustration of robust RL formulated as a zero-sum game in Appendix B (Additional Related Work) and improved our writing in sections of introduction and methodology. We believe these additions effectively address your concerns.

[1] Robust Adversarial Reinforcement Learning, Pinto et al. 2017.

[2] Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Training, HUang et al. 2021.

[3] Action Robust Reinforcement Learning and Applications in Continuous Control, Tessler et al. 2019.

Dear Reviewer GZCq,

Thank you so much for your prompt response and for improving our scores. We sincerely appreciate the valuable insights you provided for our paper, which have significantly contributed to enhancing its quality. If there are any further questions or concerns, we would be more than happy to address them.

2. GRAD shares similar idea of RARL algorithm (Pinto et al., 2017), that is, zero-sum structure to get the robust policy against the nature adversary.

This is indeed a valuable feedback. In Appendix B, we have extensively discussed other robust RL works that utilize zero-sum games and highlighted the distinctions between GRAD and these approaches.

• Converge to approximate equilibriums on an adversary population

During training, our approach achieves an approximate equilibrium on an adversary policy set, which is totally different from other robust RL papers which use the zero-sum game formulation but only learn with an adversary during alternating training. GRAD has the capability to continuously explore and learn new policies as best responses that are not present in the current policy set, which allows for a more thorough exploration of the policy space.

• Adaptable to different adversaries

Significantly, prior works, such as [2, 3, 4], primarily focus on a specific adversary or only consider the worst-case scenario, often adopting a highly pessimistic outlook that limits their adaptability to adversaries with different constraints. However, GRAD does not specifically target certain adversaries during training. Instead, at each epoch, it learns the best response to the adversary policy sampled from the adversary population, making it adaptable to different adversarial scenarios, including both temporally-coupled and non-temporally-coupled adversaries.

Furthermore, previous approaches exclusively consider specific attack domains for adversaries, such as perturbations on environmental dynamics [2, 3] and action perturbations [4]. In contrast, GRAD represents a pioneering effort by introducing the first robust RL framework that accommodates various adversaries including action, state and transition adversaries, without any constraints on attack domains. This broader scope positions GRAD as a more practical and scalable general solution for robust RL.

In summary, we appreciate the reviewer's identification of weaknesses in our related works discussion and our novelty in problem formulation, while also acknowledging our contributions. We look forward to further discussions.

[2] Robust Adversarial Reinforcement Learning, Pinto et al. 2017.

[3] Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Training, HUang et al. 2021.

[4] Action Robust Reinforcement Learning and Applications in Continuous Control, Tessler et al. 2019.

Thanks Reviewer JmX3 for acknowledging and encouraging our paper's contributions. We provide a detailed response to your insightful feedback and make detailed revisions to our paper accordingly.

1. The current framework considers robustness against state and action uncertainty. It will be better to include more detailed Related Works including and more relevant works in the revision.

We appreciate your valuable suggestion to provide a more detailed discussion on related works concerning robustness against uncertainty. Following your advice, we have expanded the discussion in Appendix B to include the mentioned papers, aiming to make our literature survey more comprehensive.

Regarding the experiments on state robustness, we present results under the most robust state adversary. Other state attack methods or state uncertainty sets are less effective in attacking both robust baselines and our approach. Furthermore, we need to clarify that our method is proposed to defend against adversarial perturbations and improve adversarial robustness, which differs from many papers that focus on uncertainty-aware RL. Adversarial RL where the learning process is intentionally disrupted by an adaptive adversary (as a max-min problem) is more challenging than uncertainty-aware RL (a max problem under uncertainty)many papers focus on uncertainty-aware RL. Remarkably, GRAD exhibits strong performance even under model uncertainty, as shown in our experiments.

We have included an additional set of experiments and updated results in Appendix C. These experiments consider model mismatch or transition uncertainty following [1], across three perturbed MuJoCo environments with changes in different physical parameters. The observed performance disparity underscores the effectiveness of GRAD and its robustness against model uncertainty.

[1] Natural Actor-Critic for Robust Reinforcement Learning with Function Approximation, Zhou et al. 2023.

Thank you for the detailed response. I do think the authors have improved the manuscript compared to the pre-rebuttal stage and also seem to have addressed the major concerns of all reviewers. I will update my score after authors-reviewers discussions.

Thanks Reviewer mjop for the insightful review of our paper. We acknowledge the concern the reviewer raised. We have updated our paper and provided a comprehensive response to address the issues highlighted.

1. Limitation of our temporally-coupled settings. Definition 3.2 only considers the temporally-coupled perturbation from the last time step. Even if the authors don't consider the general partially observable MDP, they should consider a more general case, e.g., m-order MDP [Efroni et al. 2022, Provable Reinforcement Learning with a short-term memory].

According to the reviewer's suggestion, we have expanded our experiments to include a more general scenario and introduce a memorized temporally-coupled attack. We calculate the mean of perturbations from the 10 past steps and apply the temporally-coupled constraint on this past perturbation mean. We add this experiment in Appendix C.2 to showcase the effectiveness of GRAD under this extended adversarial setting. The added results indicate that GRAD exhibits enhanced robustness compared to other robust baselines under the memorized temporally-coupled attack. We believe this addition contributes valuable insights to the paper and addresses your concerns regarding the temporal scope of perturbations.

Moreover, the primary reason we focused on perturbations from the last time step is grounded in the common practice of state adversaries, which typically perturb the current state without explicitly attacking short-term memory, but we agree that considering the short-term memory is a reasonable and considerable setting for adversaries. We eagerly anticipate any additional discussions or suggestions the review may have.

2. The zero-sum game-based approach is not new for robust training in RL, e.g., [Tessler et al., 2019].

While we acknowledge the precedent exploration of zero-sum games in robust RL within the existing literature, it is crucial to emphasize the distinctive contribution of our method, GRAD.

• Converge to approximate equilibriums on an adversary population.

During training, our approach achieves an approximate equilibrium on an adversary policy set, which is totally different from other robust RL papers which use the zero-sum game formulation but only learn with an adversary during alternating training. GRAD has the capability to continuously explore and learn new policies as best responses that are not present in the current policy set, which allows for a more thorough exploration of the policy space.

• Adaptable to different adversaries.

Significantly, prior works, such as [1, 2, 3], primarily focus on a specific adversary or only consider the worst-case scenario, often adopting a highly pessimistic outlook that limits their adaptability to adversaries with different constraints. However, GRAD does not specifically target certain adversaries during training. Instead, at each epoch, it learns the best response to the adversary policy sampled from the adversary population, making it adaptable to different adversarial scenarios, including both temporally-coupled and non-temporally-coupled adversaries.

Furthermore, previous approaches exclusively consider specific attack domains for adversaries, such as perturbations on environmental dynamics [1, 2] and action perturbations [3]. In contrast, GRAD represents a pioneering effort by introducing the first robust RL framework that accommodates various adversaries including action, state, and transition adversaries, without any constraints on attack domains. This broader scope positions GRAD as a more practical and scalable general solution for robust RL.

To provide further clarity on this aspect, we have included a detailed illustration of robust RL formulated as a zero-sum game in Appendix B (Additional Related Work) and improved our writing in sections of introduction and methodology. We believe these additions effectively address your concerns.

[1] Robust Adversarial Reinforcement Learning, Pinto et al. 2017.

[2] Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Training, HUang et al. 2021.

[3] Action Robust Reinforcement Learning and Applications in Continuous Control, Tessler et al. 2019.

3. This paper misses one classic setting of robust MDP (i.e., transition adversaries) [e.g., Iyengar'05, Robust Dynamic Programming] as well as related baselines [e.g., Zhou et al. 2023, Natural Actor-Critic for Robust Reinforcement Learning with Function Approximation].

We deeply appreciate the insightful suggestion regarding the incorporation of experiments involving transition adversaries. To attempt to study this great point, we conducted a meticulous comparative study pitting GRAD against the baseline RNAC-PPO trained with DS and IPM uncertainty sets across three perturbed MuJoCo environments with changed physic parameters, following the experimental protocols outlined in [4].

The results unequivocally demonstrate that GRAD outperforms RNAC-PPO across all tasks under transition uncertainty. This performance disparity underscores the efficacy of GRAD and its adaptability across diverse attack domains. We believe these results findings make substantial empirical contributions to the paper, effectively addressing the concern raised by the reviewer. We also add this experiment in Appendix C.2.

4. Figure 1 doesn't make sense. The robust baseline considers the worst-case scenario, which should be more stable for different kinds of attacks compared with the less conservative model that is proposed in this work.

We have revised the caption of Figure 1 to explain why the baseline is not robust to the temporally-coupled adversaries,

While it is true that WocaR-PPO aims to enhance its robustness in the worst-case scenarios or improve its lower bound of performance, it does not guarantee superiority under other case scenarios, such as temporally-coupled perturbations. The baseline's focus on worst case estimations and improvement does not inherently ensure robustness across all conceivable situations. It is noteworthy that WocaR-PPO's relatively higher risk of stumbling under temporally-coupled attacks, as demonstrated in our experiments. This is precisely where GRAD excels. GRAD finds the equilibrium with an adversary policy set, showcasing a significant advantage in diverse adversarial settings. We believe these aspects underscore the substantial advantages of GRAD and welcome further discussion on this matter.

In summary, we have responded to the reviewer's concern by providing additional experiments in temporally-coupled settings and diverse attack domains, illustrating GRAD's robustness. We also discuss our noverlty different from related works to better elucidate GRAD's strengths. Regarding Figure 1, we explained the observations and modified Figure 1's illustration in our paper. We appreciate the valuable feedback, which enhances the quality of our paper, and we look forward to further discussion and additional suggestions.

[4] Natural Actor-Critic for Robust Reinforcement Learning with Function Approximation, Zhou et al. 2023.

Thank you very much for your further response and raising a discussion-worthy question. We would like to clarify the following points.

1. Prior methods, such as RNAC, provide theoretical and empirical guarantees for robustness, focusing on uncertainty-aware RL. In contrast, our method mainly aims to enhance robustness specifically for adversarial RL, while our experiments also demonstrate GRAD's significant robustness against model uncertainty. Adversarial RL involves a max-min problem against a learnable and adaptive adversary, posing a greater challenge than uncertainty-aware RL, which is a max problem under uncertainty. Existing methods for adversarial RL often struggle to provide both theoretical guarantees and adaptability to different adversaries, often at a higher computational cost, such as alternating training with learned adversaries. The potential superiority of game-theoretic methods in adversarial RL arises from the fact that adversaries can adjust/learn their attack strategies based on the agent's policy. Hence, it becomes necessary to find the approximate equilibrium solutions to discover more robust strategies against adversarial perturbations

2. We understand your concern, and if our method were only effective with minor improvements in robustness against a limited set of uncertainties or perturbations, the extensive training time would indeed be unjustifiable. Therefore, we conducted experiments to demonstrate the broad practicality of our method against various adversarial perturbations with different constraints or in different attack domains. We are the first to defend against such a diverse range of perturbations, contributing to adversarial robustness.

3. Additionally, addressing computational efficiency, we propose in the discussion and limitations section that future directions could explore more efficient alternatives to PSRO. Our primary contribution lies in showcasing the efficacy of game-theoretic methods in robust RL, particularly against adversarial perturbations, providing a comprehensive and effective solution.

We hope our response further addresses your concerns and helps you understand the contribution of our work to the robust RL community. We welcome further discussion, and we will appropriately supplement our discussion in the Appendix.

Based on the author's efforts to solve some concerns, I have increased the score. Additionally, I don't completely agree with the authors' claim that "game-theoretic RL can be a general and scalable solution for the robust RL objective". Not only does the game-theoretic method bring instability, but it also makes training take much longer compared with the standard non-robust approach. Instead, there exist some computationally efficient methods to guarantee robustness theoretically and empirically (e.g., RNAC [4]). In other words, I am wondering if it is necessary to make slight improvements for robustness at the expense of significant training time.

To provide a clearer and more detailed explanation with further discussion:

1. Regarding Tessler et al., although their formulation considers the worst-case adversary by framing robust RL as a max-min problem, their algorithm updates the adversary using only one-step gradient descent without the guarantee of adversary convergence and makes it hard to estimate the worst-case value, contributing to their lack of robustness. In contrast, our algorithm, by finding the approximate equilibrium with the adversary set, achieves stronger robustness and is more adaptable for different adversaries.

2. We acknowledge the significance of prior works and understand that your question mainly revolves around the trade-off between robustness and computational cost. Our insights center on finding the approximate equilibrium to discover optimal robust models with higher levels of model exploitation. In many practical scenarios like self-driving or robotics, where robust models are crucial, robustness under various perturbations holds more significance than computational efficiency. This is where game-theoretic methods become valuable. In game-theoretic methods, it can also balance robustness and computational efficiency by controlling the level of exploitability.

We appreciate your continued engagement, and we believe this discussion has further clarified the value of our work. Looking forward to your response!

Dear Reviewer mjop,

Thank you once again for your response. Perhaps our previous clarification led to some misunderstandings. We want to emphasize that our main goal is to address the adversarial RL problem with an adversary that is adaptive to the agent's policy. Papers like Tessler et al. (2019) on action perturbations and Zhou et al. (2023) on model mismatch uncertainty, are hard to defend against the strongest adversarial perturbations and only empirically evaluated on uncertainty sets. We include Tessler et al. (2019) as an action-robust baseline in our paper, which is not robust under RL-based action adversaries. This vulnerability arises due to the inherent difficulty in estimating the long-term worst-case value under adaptive adversaries. Distributional robust optimization (DRO) is also challenging to apply to this challenging problem, especially against state adversaries in the high-dimensional state space. At the same time, adversarial training methods also struggle to effectively deal with model uncertainty or model mismatch problems.

Existing adversarial training methods, such as alternating training, often require simultaneous training of an RL-based adversary, leading to increased computational costs. This is worthwhile in many practical settings when we have access to a high-fidelity simulator. Our approach, leveraging game-theoretic methods, demonstrates robustness against adversarial perturbations and model uncertainty, as a more effective and general solution for high-dimensional tasks. We hope this clarifies our contributions to the robust RL community. We further elaborate on the distinctions from DRO in our Appendix.

While we fully understand your concerns, it remains challenging for existing methods to effectively address adversarial perturbations, especially when considering different attack constraints and simultaneous perturbations to both state and action spaces. GRAD, however, is applicable to different adversary settings, not limited to specific uncertainty sets or adversaries confined to certain constraints. This adaptability is a key strength that sets GRAD apart from other robust RL methods.

We greatly appreciate your valuable suggestions and hope for more discussions and considerations of your score before the end of the discussion period.

Thank you for your continued engagement and thoughtful consideration of our paper. Additional clarifications and experiments significantly contribute to the improvement of our work. Your willingness to reconsider the score for our paper is highly valued, and any further feedback you may have is welcomed.

Additionally, we will express our gratitude in the comments provided to the AC for your diligent review and discussion. We truly appreciate the time, effort, and professional insights you've invested in our discussions, which are crucial for enhancing the quality of our work.