Illusory Attacks: Information-theoretic detectability matters in adversarial attacks

Weaknesses

The paper does not seem to give much guidance on how to design better defenses against these attacks/ how can illusory attacks allow us to design more robust RL techniques?

Thank you for this remark. As stated in the Conclusion, we believe that a crucial line of defense against illusory attacks is the hardening of observation channels. We include preliminary experiments on this in Section 5 ("Robustification using reality feedback"). These experiments suggest that even the partial hardening of observation channels against adversary interference can drastically improve worst-case performance. Our paper also includes empirical evaluations of attack detectors in novel settings, for example, we embed the SOTA detector from [A] in an information-theoretic sequential hypothesis testing framework.

We now clearly point out three directions toward more robust RL techniques in the conclusion. First, research into better detection methods is required, both under the consideration of information-theoretic optimal detection approaches, as well as practical implementations that allow for real-time application. Second, robustified policies should incorporate active detection mechanisms. This would allow victims to actively seek out environment states in which the presence of adversaries may be more detectable. Third, we believe that hardening observation channels is an important part of defending against illusory attacks.

[A] Out-of-Distribution Detection for Reinforcement Learning Agents with Probabilistic Dynamics Models, Haider et al., AAMAS 2023

Questions

I'm a little confused by the definition of adversary score, what exactly is the normalization used to compute this value? The paper states that the value is normalized with respect to the "highest adversarial return in each class, as well as the victim’s expected return in the unattacked environment" but I don't understand what this means.

We provide the formula for the computation of the adversary score below. Specifically, we first compute the reduction in victim reward, normalized by the reward in the unattacked environment. Then, we normalize by the highest normalized reduction in victim reward for a specific setting (i.e. a specific environment and specific budget).

$\textrm{adversary score} = \frac{r_{\textrm{unattacked}}-r_{\textrm{attacked}}}{r_{\textrm{unattacked}}-r_{\textrm{attacked}}(\textrm{lowest for setting})}$

Weaknesses

My main concern with the paper is the presentation, which is very dense and at times difficult to understand. For instance, many of the equations in Section 4 are presented without much motivation or intuitive explanation. It seems at some points that the authors include additional mathematical details which are unconnected to the main message of the paper (e.g., discussing nonparameteric density estimation on line 188); it might be best to omit these and devote the room instead to making other points more clear. I have listed some specific problems with the writing below.

Thank you for this feedback. We have now made several updates in the revised pdf to improve Section 4, and have included additional motivations for individual definitions. We have further addressed all issues listed below. Specifically, we made the following changes:

• We have moved parts of the POMDP notation from section 3.1 to the appendix.
• We have moved details on the game-theoretic intractability of the zero-sum game $\mathcal{G}$ between the victim and the attacker to the appendix (see paragraph 3 in section 4.1).
• We have created a new section in the appendix devoted to the illusory objective estimator. We have moved our references to non-parametric density estimation (paragraph 2 in section 4.2), alongside comments on particle filtering and nested estimator bias (formerly at the end of section 4).

Additionally, I found many of the figures had only limited explanation. I don't find Figure 1 to be so clear—there is a lot going on and I could only understand it completely after going through the rest of the paper.

We have updated the caption of this figure to make it more accessible and welcome any additional feedback here. We have also included an explanatory paragraph in Section 4.

Figure 2 is presented with only high-level explanation in the paper, and it's not clear what all the arrows and lines mean.

We have redone this Figure, added additional explanations about the different graphical elements, and have also updated the caption accordingly. We are thankful for any additional feedback.

Algorithm 1 is confusing since p($\emptyset$) and p($o_{old}$,$a_{old}$) seem to be undefined and there is no captionon.

$p$ is defined in the MDP definition in the Background (Section 3.1), as the probabilistic state transition function of the MDP (conditioned on the last observation and action), while p($\emptyset$) refers to the initial state distribution. We have now clarified this further in Algorithm 1.

Specific issues:

Line 132: I believe it should be $\text{supp}\ \xi(\cdot|s)\subseteq\mathcal{B}(s)$

We believe that our notation $\text{supp}\ \xi(\cdot|s)\in\mathcal{B}(s)$ is correct, as $\mathcal{B}(s)$ is a superset of $\mathcal{S}$ (note that $\mathcal{B}(s)$ is the powerset of $\mathcal{S}$).

The definition of additive perturbations on line 134 is a bit confusing. First, why are the perturbations themselves in the state space $\mathcal{S}$. Shouldn't they be in $\mathbb{R}^{d}$ where $\mathcal{S}\subseteq\mathbb{R}^{d}$? Second, the $\delta$ notation for a point mass is not previously introduced.

Thank you for this remark. We now introduce $\delta$ in Section 3 and have fixed the clerical error in the definition of additive perturbations ($\varphi_t \in \mathbb{R}^d$).

Line 225: I believe "relative entropy" is actually another term for KL-divergence, and what is mean is "cross entropy." The notation used of  is standard for "joint entropy," though! So this is quite confusing—it would be best to clean up the notation. […]

Thank you for this remark, this was indeed a typo and we had meant to write cross entropy. We have fixed this now and also updated the notation, citing the relevant book to avoid any confusion.

In Algorithm 1,  $\hat{D}_{KL}$ is not defined.

We apologize for this mistake. We have now addressed this; $\hat{D}_{KL}$ is a trailing window estimate of the average constraint violation used to update the dual variable $\lambda$.

Questions

In Figure 5, it seems strange to the detection rate averaged across environments for human detection and automated detection when one is only evaluated on two environments and the other is evaluated on all four. Since the automated detection rate is already shown in Figure 4, why not just show the automated detection rate for the environments where human detection was tested in Figure 5?

Response: We acknowledge that the presentation was not ideal, as the limitations of the detection rate analysis were only mentioned in the bottom left corner and could be missed easily. We have now split this plot into two individual plots (one for automated and one for human detection).

Dear reviewer, we kindly wanted to ask whether we were able to address your questions and concerns.

(Minor) The paper introduces a considerable amount of notation. I feel like the main results could be conveyed with less mathematical notation, which could be moved to the appendix (a lot of basic RFL notation and concepts)

Thank you for this feedback. We have trimmed our notation without compromising on mathematical stringency. We have made several consolidations to the pdf (see details below). We welcome any further advice from the reviewer on how to make our paper more accessible.

• We have moved parts of the POMDP notation from section 3.1 to the appendix.
• We have moved details on the game-theoretic intractability of the zero-sum game $\mathcal{G}$ between the victim and the attacker to the appendix (see paragraph 3 in section 4.1).
• We have created a new section in the appendix devoted to the illusory objective estimator. We have moved our references to non-parametric density estimation (paragraph 2 in section 4.2), alongside comments on particle filtering and nested estimator bias (formerly at the end of section 4) to this section.

(Minor) The evaluation is limited to simple simulated environments

We would like to point out that we evaluate on standard benchmarks in the field [A, B, C]. We however agree with the reviewer that future work should investigate more complex environments, as we also state in our conclusion.

(Medium) I'm missing a runtime analysis to compare the efficiency of the different attacks. This would also highlight if the framework could be scaled to more complicated problems

Thank you for this remark. We have added an empirical runtime analysis to Appendix A.11. The required wall clock time for training illusory attacks is about $1.6$ times higher than for training standard adversarial attacks (SA-MDP attacks). We acknowledge that this is a significant, albeit not prohibitive, increase in wall clock time. At test-time, inference times for illusory attacks are identical to existing attacks as they only consist of a neural network forward pass. Memory requirements are identical. We found that wall clock time is generally higher in the more complex Hopper and HalfCheetah environments, as compared to the CartPole and Pendulum environments.

Citations:
[A] "Robust reinforcement learning on state observations with learned optimal adversary", Zhang, 2021
[B] "Policy Smoothing for Provably Robust Reinforcement Learning", Kumar et al., ICLR 2022
[C] "Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learning", Liang et al., NeurIPS 2022

Thank you for your comment, we have carefully re-evaluated all three papers. We respectfully disagree, and do not find that these works require rephrasing our claims, as these works do not affect our contributions. Further, we do not agree that a comparison to [1] is required.

[1] (Korkmaz et al. 2023): We agree that this paper is concerned with detecting adversarial attacks on deep RL agents. Our paper employs detectors in order to evaluate novel information-theoretic attacks (illusory attacks). We decided not to employ the proposed detector in [1] for two reasons. The detector in [1] is restricted to discrete action spaces, and it does not seem trivial to extend it to continuous action spaces. Furthermore, as our attacks are information-theoretically motivated, we believe that comparison against information-theoretically motivated detectors is more adequate. We will add [1] to our related work section.
[2] (Korkmaz 2023): This paper proposes adversarial attacks motivated by a notion of imperceptibility measured in policy network activation space. One major difference is that the paper focuses on per-state imperceptibility, while our work focuses on information-theoretic undetectability, which hence requires focusing on whole trajectories. We will add [2] to our related work section.
[3] (Korkmaz 2022): This paper finds that neural policies learn non-robust features that are shared across baseline deep reinforcement learning training environments. This direction of research seems only remotely related to our paper.

We would also like to point out that we cite a number of influential works on robustification in the sequential MNP framework (see Section 2 in our paper), amounting to more than 100 total citations. This includes works on randomized smoothing, information-theoretic detection methods, test-time hardening by computing confidence bounds, training with adversarial loss functions, and co-training with adversarial agents. We therefore believe this related space to be well-represented.