Diffusion Guided Adversarial State Perturbations in Reinforcement Learning

Dear Reviewer D5uT:

Thank you for your helpful and insightful feedback. We will address your concerns point by point in the following.

Q1: Can the reconstruction error effectively represent the realism of state perturbation?

A1: Thank you for this important comment. As suggested by the reviewer, we have created two gifs, one showing the perturbed trajectory under our attacks constrained by the realism detection, and one showing the unperturbed trajectory for comparison and uploaded them in the updated supplementary material.

We have further explored the Wasserstein-1 distance between a perturbed state and the true state of the previous time step as another realism metric shown in Table 2-2 in the general response. As argued in [1], the Wasserstein distance captures the cost of moving pixel mass and represents image manipulation more naturally than the $l_p$ distance. Thus a small Wasserstein distance from previous true states shows that even when the agent is aware of the true previous state $s_{t-1}$, the perturbed state $\tilde{s}_t$ generated by our attack is more stealthy than other attacks. The results shown in Table 2-2 state that our attack method achieves both the lowest Wasserstein distance and the lowest reconstruction error. This experiment further proves that our attack method can generate realistic attacks and achieve superb attack performance at the same time.

Q2: More empirical studies are needed such as ablation on varying attack strength.

A2: We included the ablation study on varying guidance strength in Appendix E in the original submission, where we also included ablation studies on effectiveness of the realism guidance and target action selection methods. We agree with the reviewer that more evaluation results would help prove our attack method’s effectiveness and soundness. Thus, we have conducted new experiments including (1) an experiment for the Atari Roadrunner environment (see Table 1 in the general response), (2) an experiment that compares different attack methods including the newly added PA-AD attack (see Table 2 in the general response), and (3) an experiment that compares the DDPM and EDM diffusion methods (see Table 5 in the general response). We hope these additional ablation studies can address your concern.

Q3: Use PA-AD or other non-myopic target action selection methods.

A3: We would like to thank the reviewer for this insightful comment. Combining PA-AD or other non-myopic target action selection methods with our conditional diffusion based attack is indeed a promising direction. A simple strategy is to adopt a two-step approach, similar to what we did in our paper, where one first applies a non-myopic target action selection method to select target actions with some long-term attack objective in mind, and then uses our diffusion-based attack to approximate the goal. An important challenge, however, is that a diffusion-based approach cannot guarantee that the target actions are alway chosen both due to the randomness of the diffusion model and the requirement for maintaining realism and history alignment. Therefore, to provide a performance guarantee, one needs to measure the expected success rate of a conditional diffusion model for a given set of target actions, which are then optimized toward some long-term goal that is achievable. We believe this requires a non-trivial extension of our approach and is an interesting future direction to explore.

We sincerely hope our responses and additional experiment results have addressed all your concerns. If so, we kindly hope that you consider increasing the rating of our paper. Please do not hesitate to add further comments if you have any additional questions or need further clarifications.

[1] Eric Wong, Frank R. Schmidt, and J. Zico Kolter. Wasserstein Adversarial Examples via Projected Sinkhorn Iterations. ICML 2019.

In conclusion, although both our attack method and [1] attempt to go beyond $l_p$ norm bounded attacks, the attack methods in [1] focus on policy independent attacks that are visually imperceptible and can compromise defenses like SA-DQN but these attacks cannot change the essential semantics of image observations and fail to fight against diffusion-based defenses. In contrast, our attack method used a conditional diffusion model to generate perturbed states that can change the essential semantics of the image data while remaining stealthy to bypass the strong defenses including diffusion-based defenses.

Q2: The submission substantially lacks appropriate references. The claimed contributions of the submission are misplaced and incorrect.

A2: We thank the reviewer for the comment and for pointing out the missing related work. After careful review, we found that seven of the nine papers mentioned by the reviewer were included in our original submission. The remaining two papers [1] [2] both focus on high-sensitivity direction attacks, where the approaches are significantly different from ours as discussed above. We have carefully revised our paper to acknowledge their contributions and highlight the novelty of our work. We have also included a discussion of these papers in the related work section of the revised submission. As the code in [1] and [2] are not publicly available, we were not able to include the detection method in [2] as a baseline, but we have managed to implement the attack methods in [1] and showed that they cannot compromise diffusion-based defenses as discussed above.

Q3: Lack of technical and experiment details for reproducibility.

A3: We discussed our experiment setting in Appendix D.6 in the original submission, where we reported the hyperparameters for training the conditional diffusion model and the guidance strength during testing. As we mentioned there, we used the default parameters setting in their original papers for other defense methods. We also provided the source code together with our paper during submission. In response to the reviewer’s concern, we have added a section on pre-processing the Atari environments for better reproducibility. In the revision, we have provided a set of new ablation study results, as discussed in the general response. For each of them, we have provided some insights into the results and connected them with our approach in Section 3. For example, we have provided a detailed comparison of our attack with the high-sensitivity direction attacks in [1] and highlighted the importance of considering both static and dynamic stealthiness in deep reinforcement learning with image input. We have also adapted the Wasserstein distance perturbation metric [3] originally proposed for adversarial examples in deep learning to the deep reinforcement learning context by considering the sequential decision-making nature of RL. We hope these efforts adequately address the reviewer's concern about the lack of technical details.

Q4: Missing Roadrunner results.

A4: As suggested by the reviewer, we have added new experiment results on Roadrunner in Table 1 in the general response. We have retrained the vanilla DQN model and the SA-DQN model because the pretrained vanilla DQN and SA-DQN models provided by the SA-MDP paper [4] do not work in the RoadRunner environment. The result shows that similar to other Atari environments we evaluated before, our attack is able to compromise SA-DQN and Diffusion History defenses in the Roadrunner environment.

Q5: Improper use of the word “poisoning.”

A5: We carefully reviewed the paper and found that we misused the word “poisoning” once. We thank the reviewer for catching this and have corrected it in the revision.

We sincerely hope our responses and additional experiment results have addressed all your concerns. If so, we kindly hope that you consider increasing the rating of our paper. Please do not hesitate to add further comments if you have any additional questions or need further clarification.

[1] Ezgi Korkmaz, Adversarial Robust Deep Reinforcement Learning Requires Redefining Robustness, AAAI 2023.

[2] Ezgi Korkmaz and Jonah Brown-Cohen, Detecting Adversarial Directions in Deep Reinforcement Learning to Make Robust Decisions, ICML 2023.

[3] Eric Wong, Frank R. Schmidt, and J. Zico Kolter, Wasserstein Adversarial Examples via Projected Sinkhorn Iterations, ICML 2019.

[4] Huan Zhang et al., Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations, NeurIPS 2020.

Dear Reviewer 37tk:

Thank you for your insightful feedback. We will address all your concerns point by point in the following.

Q1: The main claimed contributions have been mentioned and analyzed in [1].

A1: We would like to thank the reviewer for pointing us to this important study [1]. We have updated the introduction section of our paper to acknowledge the contribution of [1] and also included a detailed comparison with [1] in the related work section and the evaluation section. However, after carefully comparing our work with [1] and conducting additional environments (as the code in [1] is not publicly available, we have tried to reproduce some of their results in our environment), we found that although both [1] and our work consider attacks beyond $l_p$ norm constraint, the two attack methods are significantly different in multiple aspects and our paper has made substantial new contributions beyond what is already considered in [1].

First, our attack is able to change the essential semantics that matter for decision making, making it much harder to defend. [1] shows that by following high-sensitivity directions, including changing brightness and contrast, image blurring, image rotation and image shifting, it is possible to generate perturbations that are visually imperceptible and semantically different from the original state. These attack methods reveal the brittleness of robust RL methods such as SA-DQN, but they mainly target changes in visually significant but non-essential semantics. For example, the relative distance between the pong ball and the pad will remain the same after brightness and contrast changes or image shifting in the Pong environment. Consequently, the perturbed images generated by these methods can potentially be denoised by a diffusion model. To confirm this, we have conducted new experiments, showing that (1) the Diffusion History defense with a diffusion model trained from clean data only is able to defend against B&C, blurring, and small scale rotation and shifting attacks (see Table 4-1 in the general response), and (2) when the diffusion model is fine-tuned by randomly applying image rotations or shifting during training, the Diffusion History defense can mitigate large scale image rotations and shifting considered in their paper (see Table 4-2 in the general response). In contrast, our diffusion guided attack can change the decision-relevant semantics of images, such as moving the Pong ball to a different position without changing other elements in the Pong environment as shown in Figure 1 e) in the paper. This is the key reason why our attack can bypass strong diffusion based defense methods.

Second, our attack is stealthy from both static and dynamic perspectives. [1] claims that the perturbed states generated by their high-sensitivity direction based attacks are imperceptible by comparing the perturbed state $\tilde{s_t}$ and the true state $s_t$. However, we found that this only holds for small perturbations. For example, the Rotation attack with degree 3 and Shifting attack (1,2) in the Pong environment considered in their paper can be easily detected by humans (see Figure 6 in Appendix E in our revised paper). Further, their metric for stealthiness is static and does not consider the sequential decision-making nature of RL. In contrast, our attack method aims to stay close to the set of true states $S^*$ to maintain static stealthiness (Definitions 1 and 2 in our paper) and align with the history to achieve dynamic stealthiness (Definitions 4 and 5). These are novel definitions for characterizing stealthiness in the RL context. The static stealthiness is demonstrated through the low reconstruction loss of the perturbed states generated by our method shown in Figure 3 a) in our paper. To better illustrate that our attacks are stealthy from a dynamic perspective, we have added an ablation study to compare the average Wasserstein-1 distance between a perturbed state and the previous step’s true state across a randomly sampled episode (see Tables 2-2 and 4-3 in the general response). As argued in [3], the Wasserstein distance captures the cost of moving pixel mass and represents image manipulation more naturally than the $l_p$ distance. The results show that even when the agent is aware of the true previous state $s_{t-1}$, the perturbed state $\tilde{s}_t$ generated by our attack is more stealthy than other attacks including the attack methods in [1] (Table 4-3 in the general response).

(Continue in the next comment)

Dear Reviewer 37tk:

Thanks for your feedback on our rebuttal and we will provide further clarifications on your concerns.

Q1: [1]'s methods can change the essential semantics and decrease 90% of the policy's performance without accessing the victim's policy.

A1: We acknowledge that rotations and perspective transformations in [1] may alter the absolute distances, such as between the Pong ball and the paddle. However, it is crucial to note that these changes do not affect the semantic meaning based on our Definition 3: after projecting perturbed states onto the true state set, the projected states remain close to the original states. This explains why diffusion-based defenses effectively counter the attacks in [1], as shown in Tables 4-1 and 4-2 in the general rebuttal, by recovering true states from perturbed ones.

Additionally, we note that large-scale transformations in [1] are not stealthy. For example, we provide images of 3-degree rotations in Appendix E, where the perturbed states are visually distinguishable from true states, violating static stealthiness.

Q2: Authors currently still missing quite critical issues that are essential to adversarial machine learning.

A2: We agree that the policy-independent and history-ignorant attack given in [1] is lightweight to implement and opens up an interesting direction. However, it also has fundamental limitations. First, it cannot bypass diffusion-based adversarial training and may generate unrealistic perturbations, as discussed above and shown in the revised paper. Second and more importantly, it largely ignores the sequential decision-making nature of RL. All the attacks implemented in [1] are applied to individual states in a myopic way. Thus, they either do not change the semantics essential to decision-making (e.g., when the same rotation operation is applied to all the states as in the paper) or cannot generate history-consistent perturbations (e.g., when different operations are applied in each state). Our approach aims to address these limitations.

Q3: Our work is the first to claim an attack beyond $l_p$-norm.

A3: We would like to clarify that we did not claim to be the first to propose attacks beyond $l_p$-norm in our paper. Furthermore, we have cited [1] in our revised manuscript and explicitly identified it as a beyond $l_p$-norm attack in the introduction, related work, and evaluation sections. We will further clarify this by incorporating the discussion above into the revision.

Q4: The statement "adversarial training is not effective against beyond $l_p$-norm attacks" is proved in previous work.

A4: We respectfully disagree with the reviewer on this. Before our work, it was unclear if there were any adversarial perturbation attacks, including those beyond $l_p$-norm constraints, that could bypass all existing robust RL approaches while remaining stealthy. As shown in Tables 4-1 and 4-2, a diffusion model trained with adversarial data generated by the attacks in [1] can successfully defend against these beyond $l_p$-norm attacks in [1]. This demonstrates that adversarial training can be effective against such attacks.

Furthermore, we only claim that our proposed attack compromises existing robust RL methods. We do not rule out the possibility that novel adversarial training-based defenses could effectively counter our attack.

We hope our clarifications address your concerns, and we welcome any further feedback.

Dear Reviewer eNaA:

Thank you for your valuable and insightful feedback on our work. We will address your questions and concerns point by point.

Q1: Motivation of changing the semantics.

A1: As shown in recent work such as DP-DQN [3] and Diffusion History (inspired by [5]), current $l_p$ norm bounded attacks including PA-AD cannot bypass these diffusion-based defense methods in environments with raw-pixel inputs such as Atari games even with a relatively large perturbation budget (please refer to Table 2-1 in the general response). We argue that the main reason is that these attacks are not able to change the essential semantics of the image input under a reasonable attack budget, so that a diffusion-based defense can purify the noise injected to gain strong defense performance.

The existence of these strong diffusion-based defenses against $l_p$ norm bounded attacks motivate us to develop new attacks that can change the semantics of states to mislead those defense methods to choose non-optimal actions even after purifying the perturbed states. Our attack still remains stealthy from both a static and a dynamic perspective by utilizing a history conditioned diffusion model with realism guidance. The static stealthiness is demonstrated through the low reconstruction loss of the perturbed states generated by our method shown in Figure 3 a) in our paper. To better illustrate that our attacks are stealthy from a dynamic perspective, we have added an ablation study to compare the Wasserstein-1 distance between a perturbed state and the true state in the previous time step. As shown in the general response Table 2-2, our attack method has the lowest average Wasserstein distance among all the attacks. As argued in [4], the Wasserstein distance captures the cost of moving pixel mass and represents image manipulation more naturally than the $l_p$ distance. The result shows that even when the agent is aware of the true previous state $s_{t-1}$, the perturbed state $\tilde{s}_t$ generated by our attack is more stealthy than other attacks.

Q2: Recent defense baselines in [1] and [2].

A2: We thank the reviewer for pointing us to these studies. We have included them in the related work section of the revised submission. However, after reading both papers carefully, we found that the defense methods proposed in [1] and [2] might not directly apply to our setting due to the following reasons. First, both studies conduct their experiments in MuJoCo environments where the state spaces are much smaller compared with Atari environments with image input. Further, these approaches are already computationally expensive (both take more than 20 hours) to train in MuJoCo environments. Thus, directly applying them to image domains can be computationally prohibitive, which points to an interesting research direction for further study. Second, the code for [1] is not publicly available at this time and the code for [2] only implements MuJoCo environments so we cannot easily evaluate these two defense methods against our attacks in Atari environments.

Although we were not able to implement the game theoretic defense in [1] due to the lack of code and the expected high computational overhead in Atari environments, we would like to point out that the DP-DQN [3] defense currently considered in the paper also adopts a game-theoretic approach by identifying an approximate Stackelberg equilibrium. While the vanilla DP-DQN uses PGD attack to simulate worst case attacks, we have retrained DP-DQN by replacing the bounded PGD attack with our unbounded diffusion guided attack. We trained this modified DP-DQN on the Pong environment with 1 million steps but the reward remained at -21 (the worst case) all the time. This gives evidence that even game theoretical defenses might not be strong enough to defend against our attacks.

We sincerely hope our responses have addressed all your concerns. If so, we kindly hope that you consider increasing the rating of our paper. Please do not hesitate to add further comments if you have any additional questions or need further clarifications.

[1] Liang et al., Game-Theoretic Robust Reinforcement Learning Handles Temporally-Coupled Perturbations. ICLR 2024

[2] Liu et al., Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policies. ICLR 2024

[3] Xiaolin Sun and Zizhan Zheng, Belief-Enriched Pessimistic Q-Learning against Adversarial State Perturbations. ICLR 2024

[4] Eric Wong, Frank R. Schmidt, and J. Zico Kolter. Wasserstein Adversarial Examples via Projected Sinkhorn Iterations. ICML 2019.

[5] Zhihe Yang and Yunjian Xu, DMBP: Diffusion model-based predictor for robust offline reinforcement learning against state observation perturbations. ICLR 2024

Q5: Lack of discussion on [1]

A5: We thank the reviewer for pointing us to this important study and we have added it to the related work in the updated version of our paper. However, after carefully reading the paper, we found that the temporally coupled attack and the game theoretic defense method proposed in [1] might not directly apply to our setting due to the following reasons.

First, all the experiments in [1] are conducted in MuJoCo environments, where the state spaces are much smaller compared with Atari environments with image input. Further, their approaches are already computationally expensive (both take more than 20 hours) to train in MuJoCo environments. Thus, directly applying them to image domains can be computationally prohibitive, which points to an interesting research direction for further study. Second, the code for [1] is not publicly available at this time so we cannot easily evaluate their attacks and defenses as baselines in Atari environments.

In response to the reviewer’s concern, we have implemented a PGD version of the temporally coupled attack in Atari environments and tested it against SA-DQN and DP-DQN. The results are in the general response Table 3. The results show that the temporally coupled PGD attack with $\epsilon = 15/255$ and $\bar{\epsilon} = 7.5/255$ could compromise SA-DQN but not diffusion based defense DP-DQN even with a large perturbation budget, which indicates the challenge of adapting this attack to Atari environments with raw-pixel input. We conjecture that this is because the attack is still constrained by an $l_p$ norm bound, making it difficult to alter the essential semantics of image input.

We sincerely hope our responses and additional experiment results have addressed all your concerns. If so, we kindly hope that you consider increasing the rating of our paper. Please do not hesitate to add further comments if you have any additional questions or need further clarifications.

[1] Liang et al., Game-Theoretic Robust Reinforcement Learning Handles Temporally-Coupled Perturbations, ICLR 2024

[2] Sun, Y., Zheng, R., Liang, Y., & Huang, F. Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RL. ICLR 2022.

[3] Eric Wong, Frank R. Schmidt, and J. Zico Kolter. Wasserstein Adversarial Examples via Projected Sinkhorn Iterations. ICML 2019.

Dear Reviewer L64s:

Thank you for your insightful feedback on our works. We will address your concerns point by point in the following by providing additional results and clarifications.

Q1: Lack of computational costs analysis of our attack.

A1: We are delighted to provide detailed computational costs of our attacks here. During the training stage, it takes around 1.5 hours to train both the conditional diffusion model and the autoencoder based realism detector. We remark that these two components can be trained in parallel. During the testing stage, our attack takes around 0.2 seconds to generate a perturbed state, making it feasible for real-time attacks.

Q2: Lack of comparison between DDPM and EDM diffusion architectures.

A2: We have provided new results to compare DDPM and EDM in terms of attack efficiency and computational cost in the general response (Table 5). The results show that EDM and DDPM exhibit similar attack performance. However, DDPM is significantly slower than EDM in terms of running time (the average time needed to generate a single perturbed state during testing), making DDPM incapable of generating real-time attacks during testing. This validates the selection of EDM as the diffusion model architecture for constructing our attacks.

Q3: Our methods were only tested on three Atari environments.

A3: We have included new evaluation results on the RoadRunner environment that is widely used in previous work. Please see Table 1 in the general response, which shows that our attack obtains superb performance against SA-DQN and Diffusion History defenses.

Q4: Comparison with other attack baselines is insufficient.

A4: We have included evaluation results for a new attack baseline, PA-AD [2], which is considered one of the strongest attacks in the literature. A complete comparison between PGD, MinBest, PA-AD and our attack methods is given in Table 2-1 in the general response, which reports both the reward and the deviation rate of each method. We want to emphasize that while the manipulation rate does not apply to PGD, MinBest and PA-AD, the reward and the deviation rate (the fraction of the chosen actions under perturbed states differ from the actions under the true states across an episode) do apply to all the baselines. The results in Table 2-1 show that our attack method achieves the best attack performance against both SA-DQN and DP-DQN in terms of both reward and deviation rate.

Furthermore, we have added the Wasserstein-1 distance between a perturbed state and the true state in the previous time step as a new metric to measure the dynamic stealthiness of baseline attacks. As argued in [3], the Wasserstein distance captures the cost of moving pixel mass and represents image manipulation more naturally than the $l_p$ distance. We have reported the reconstruction loss of perturbed states and the Wasserstein distance between a perturbed state and the previous step’s true state in Table 2-2 in the general response. The former captures the static stealthiness while the later captures the dynamic stealthiness as we further elaborated in the general response. The results show that our attack achieves both the lowest reconstruction error and the lowest Wasserstein distance, indicating our attack method achieves best stealthiness from both static and the dynamic perspectives. The superb attack performance and stealthiness of our method justify the use of the conditional diffusion model to generate attacks.

Dear Reviewer L64s:

Thank you for reviewing our rebuttal and for increasing your rating. In response to your valuable feedback, we have further revised our manuscript to include additional experiments conducted during the rebuttal period. The following changes have been made to the main text:

1. The RoadRunner results have been added to Table 1.
2. Figure 3 now includes results for PA-AD, temporally coupled, and high-sensitivity direction based attacks, along with the introduction of Wasserstein distance as an additional metric to measure stealthiness.
3. We have added the ablation study on DDPM and EDM diffusion architectures.

We believe that these revisions, along with the extra experiments and discussions, strengthen the paper's statements and conclusions.

We appreciate your continued feedback and hope that the revised manuscript addresses your concerns effectively.

Table 3 Temporally Coupled Attack on Atari

In response to Reviewers eNaA and L64s, we have implemented the temporally coupled attack proposed in [4] on top of PGD (as their code is not publicly available) and evaluated the PGD-based temporally coupled attack on Atari Pong with $\epsilon = 15/255$ and $\bar{\epsilon} = 7.5/255$. This table shows that the temporally coupled attack can compromise SA-DQN but not diffusion-based DP-DQN defense even with a large perturbation budget, indicating the challenge of adapting this attack to Atari games with raw-pixel input. We conjecture that this is because the attack is still constrained by an $l_p$ norm bound, making it difficult to alter the essential semantics of image input.

Table 4-1 Pretrained Diffusion History defense against attacks in [1]

In response to Reviewer 37tk, we have implemented the four high-sensitivity direction attacks in [1] (as their code is not publicly available), and conducted new experiments to evaluate how these attacks perform under diffusion based defenses for Atari Pong and Freeway. This table shows that the Diffusion History defense with a diffusion model trained on clean data only is able to defeat the Brightness&Contrast and Blurred Observation attacks, as well as Rotation and Shifting attacks with small rotations and shifts, although it is ineffective against large rotations (>1 degree) and shifts used in [1].

Table 4-2 Fine-tuned Diffusion History against attacks in [1]

We further fine-tuned a diffusion model by randomly applying rotations and shifts to the game frames during training, where the rotation degree is randomly chosen between 0 and 3, and the shift magnitude is randomly chosen between (0,0) and (3,3). This table shows that the fine-tuned Diffusion History defense can successfully mitigate both Rotation and Shifting attacks, even under relatively large rotations and shifts considered in [1]. In contrast, the Diffusion History defense is ineffective against our policy-adaptive attack.

Table 4-3 Wasserstein Distances of attacks in [1] and ours

This table compares the average Wasserstein Distance between a perturbed state and the previous step’s true state across an episode, under the attack methods in [1] and our attack. The results show that our attack method has the lowest Wasserstein distance compared with the four attacks evaluated in [1], indicating that our attack is more stealthy.

Table 5 DDPM vs. EDM

As recommended by Reviewer L64s, we have compared DDPM and EDM in terms of effectiveness and efficiency. The results in the table show that EDM and DDPM exhibit similar attack performance. However, DDPM is significantly slower than EDM in terms of running time (the average time needed to generate a single perturbed state during testing), making DDPM incapable of generating real-time attacks during testing. This validates the selection of EDM as the diffusion model for constructing our attacks.

[1] Ezgi Korkmaz. Adversarial Robust Deep Reinforcement Learning Requires Redefining Robustness. AAAI 2023.

[2] Sun, Y., Zheng, R., Liang, Y., & Huang, F. Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RL. ICLR 2022.

[3] Eric Wong, Frank R. Schmidt, and J. Zico Kolter. Wasserstein Adversarial Examples via Projected Sinkhorn Iterations. ICML 2019.

[4] Liang et al., Game-Theoretic Robust Reinforcement Learning Handles Temporally-Coupled Perturbations. ICLR 2024