On Minimizing Adversarial Counterfactual Error in Adversarial Reinforcement Learning

We sincerely thank the reviewer for their valuable comments and insights. Below, we address the points raised in the review.

1. Do we care about unperturbed performance since basically all real-world applications have states being perturbed? To answer this, we should highlight the difference between adversarial perturbations and non-adversarial or stochastic perturbations. The latter case can be present in the training environment and is accounted for in the expected value of a policy. To this end, PPO is generally sufficient. We include the adversarially unperturbed case (which some call the "natural" reward) in our experiments to showcase the trade-off between adversarially-unperturbed and adversarially-perturbed rewards for each algorithm.

2. Both A2B and A3B require KL divergence between policies, which should be computational costly. We would like to clarify a misunderstanding here: our method computes the KL-Divergence between two probability distributions over actions in a state and not the entire policies, thus that step is not computationally costly.

3. Can you provide any intuition about algorithmically, why this method can have such good empirical performance? Intuitively, the identification of the adversarial problem as a POMDP allows our methods to assign appropriately cautious actions by forming beliefs about the true state. Our belief construction puts more probability mass on states that are most likely given adversarial intent and reduces unnecessary focus on unlikely perturbations.

4. the authors claim that $\delta$ is no longer dependent on $b$, what's the purpose of $b(s)$ in A2B and A3B? Regarding Eq. 2: We state that C-ACoE $\delta$ is a function of $s_o$ only, since just above that equation we state that $b(s_o)$ is also a function of $s_o$ only, that is, $\delta(s_o, b(s_o))$ can be viewed as a function of $s_o$ only as the second input $b$ can be constructed completely from $s_o$. This is also why in the RHS of Eq. 2 we can write $b(s_o)$, that is, the belief as a function of input $s_o$. These belief functions are later given by A2B/A3B.

We will update the PDF with our new results and typo fixes after performing all experiments asked by all reviewers.

We sincerely thank the reviewer for their valuable comments and insights. Below, we address the points raised in the review.

1. Analysis of the trade-off between natural performance and adversarial robustness: To check the tradeoff between natural and robust performance, we can examine the role of the $\lambda$ parameter, which dictates how value-seeking or safety-seeking the agent is. This term is also present in all baselines we compare to with the same value, $0.2$, in all domains. Below we provide a sensitivity analysis for this parameter in Table 3.1 for A3B. However, it would take some time to retrain each method for a full comparison between all baselines.

2. Extension to high-dimensionality continuous domains: We request the reviewer to refer to lines 365-373 on how we handle continuous domains.; We also note that the Atari domains have images as inputs, which are considered continuous data points and are extremely high dimensional (86863*255).

3. Sensitivity analyses on key hyperparameters: Please see Tables 3.1 and 3.2 below for experiments on $\lambda$ and neighborhood size. We observe that for $\lambda$ close to 0.2, the performance is quite stable. The same goes for neighborhood sample size.

Table 3.1 $\lambda$ variations in Halfcheetah

Table 3.2 Neighborhood Sample Size Variations in Halfcheetah

4. Surrogate belief construction choices.: We propose two belief alternatives, A2B and A3B, that work in an unperturbed training environment. As far as we know, we are the first to suggest such surrogate beliefs, and we are not aware of other alternatives in the literature. We would be happy to compare to other belief constructions that the reviewer has in mind.

We will update the PDF with our new results and typo fixes after performing all experiments asked by all reviewers.

We sincerely thank the reviewer for their valuable comments and insights. Below, we address the points raised in the review.

1. Both A2B and A3B rely on some knowledge of the attacker, such as attack radius and attack preference. Can the authors provide more justification on whether this is a realistic assumption?

To answer the reviewer’s question in two parts:

1a) the attack radius $\epsilon$ is often assumed in literature to be a bound outside which an attack is considered "detected" and subsequently disregarded or heuristically intervened upon. This is an assumption utilized in most adversarial ML and adversarial RL research. Utilizing unbounded $\epsilon$ would correspond empirically to converting a stop sign to a closed road and hence is not realistic/justified.

1b) In our paper, we assume an adversary is operating to reduce the reward of the victim agent. We find this to be an intuitive formulation, as this fits the widely followed zero-sum formulation of adversarial RL.

2. How are attack parameters chosen in different scenarios? We use the same attack radius for all environments $\epsilon=0.1$ (which is the same as prior methods [1,2]) in training, and $\epsilon\geq 0.15$ in attack testing. The surrogate attack we use is PGD for all training, chosen for consistency and theoretical soundness.

3. How does performance change when the surrogate and actual attack constraints are different? If the attacker is more constrained than expected (in terms of $\epsilon$), the performance of any robust method can be expected to improve. On the other hand, if the attacker is less constrained, our method experiences less performance degradation than SOTA (see Table 2.2). Regarding different attacker preferences, we can refer to Table 2.1.

We also provide a quick comparison to an adversary that maximizes the victim reward as an example of a different opponent preference in Table 2.1. We observe that the performance of both our approach and the existing SOTA approach improves with such an adversary in place.

[1] Yongyuan Liang, Yanchao Sun, Ruijie Zheng, and Furong Huang. Efficient adversarial training without attacking: Worst-case-aware robust reinforcement learning. 2022

[2] Tuomas Oikarinen,Wang Zhang,Alexandre Megretski,Luca Daniel,and Tsui-Wei Weng. Robust deep reinforcement learning through adversarial loss. 2021

Table 2.1 Alternate attacker preference example

Table 2.2 Attacker constraint analysis

We will update the PDF with our new results and typo fixes after performing all experiments asked by all reviewers.

We sincerely thank the reviewer for their valuable comments and insights. Below, we address the points raised in the review.

1. Tradeoffs for using a history: Including a history can give our algorithm a more informed estimate of the probability of the true state given observations. In Appendix D of the submission, we derive variants of A2B and A3B that work for a history of two observations. As can be seen from that derivation, if the past observations are assumed to be perturbations of true states as well, the belief computation scales combinatorially with the length of the history. On the other hand, if the history states are taken as ground truth, there is no change in computational complexity. In either case, there is no change in complexity at run time (a forward call of the network). Empirically, adding a one-state LSTM history to our model increases training time on Mujoco-Halfcheetah from $\sim 800$ episodes (4.5 hours) to $>3000$ episodes (16 hours) to obtain sufficient samples for the belief. However, there is little tangible benefit in doing so, as seen in Appendix Table 9 and Table 1.1 below.

Table 1.1: A3B-LSTM on HalfCheetah

2. Algorithm-adaptive attackers: Our approach A3C performs well against PA-AD adversaries and PA-AD attacks work in the white-box setting where the victim policy network is available to the adversary. As far as we know, we have not come across an attack where the adversary exploits knowledge of the underlying algorithm (more than white-box; open-box?). We believe that this would make for exciting future work, not only in this setting but in general adversarial machine learning as well.

3. Other RL Adversaries: In the space of online RL, there are two broad formulations of attacks: perturbations to observations, and perturbations to the environment (simulator) directly, such as perturbations to the transition function. Our method targets the first case, as the central tenet of ACoE is recognizing and addressing the mismatch between the observed and true state, i.e., the adversary does not modify the simulator directly. As the adversary can directly modify the simulator in the second case, our method isn’t directly applicable to such attacks. However, in a similar vein to our work, there is a possible approach of forming beliefs about the adversary’s modification of the simulator and acting accordingly in the context of environment attacks.

We will update the PDF with our new results and typo fixes after performing all experiments asked by all reviewers.

Dear Reviewers,

Thank you once again for your valuable time and feedback. In the spirit of the extended discussion timeline, we have posted a revised version of the submission PDF including the following suggestions, clarifications, and analyses:

Table 9: We have included a comparison of LSTM and single-state ACoE models on continuous action domains to the discrete action domains we had submitted.

• Tables 10 & 11: We have added ablation studies for hyperparameter sensitivities during training.

• Section E.3: We have included a detailed discussion of Tables 9, 10, & 11 to the Appendix.

• Eq 2: We have updated the equations to fix the missing symbols.

• Line 362: we have revised the section to resolve notational errors.

We hope the reviewers appreciate the novelty of our formulation and solution to the adversarially perturbed state problem in RL using the partially observable nature of the problem—we believe this is an impactful development in the area of adversarially robust RL. We hope the reviewers are satisfied with the clarifications made and would be happy to engage in further discussion.