SHINE: Shielding Backdoors in Deep Reinforcement Learning

We thank the reviewer for the constructive comments. Please see below for our responses.

1. About the technical contributions.

Thank the reviewer for the comment. We agree with the reviewer that our method indeed uses the explanation method proposed by Guo et al., 2021b as it is the state-of-the-art step-level explanation method. However, we would like to respectfully argue that this does not dilute our method’s technical contribution. First, the overall defense framework that is composed of explanation-driven trigger restoration and retraining-based shielding is novel and is first proposed in this work. Second, rather than the step-level explanation, all the other technical components are newly proposed in this work. As such, we propose a new defense framework for backdoor defense with only one component leveraging an existing method. It is common that a newly proposed technique uses some state-of-the-art methods as part of its framework. We believe the proposed technique still demonstrates decent technical contributions.

2. About the proposed method requiring failure trajectories.

Thank the reviewer for the comment. As shown in the response to Review r1tp, we demonstrate that our method can work with only 20 failure trajectories. In addition, as stated in Section 5-Postmortem defense, we discuss that our goal is to quickly react to the backdoor attack once it is triggered. This is a common setup even for security-critical domains. For example, in cybersecurity, there is a directly called postmortem program analysis, which analyzes the reason and fixes the vulnerabilities after it is triggered. As is also discussed in Section 3.1, in practice, it is also difficult to guarantee the environment is clean. For example, in self-driving cars, it is extremely hard to guarantee that the attacker cannot access the environment where the car is deployed. As such, it is possible that the attacker will trigger a backdoor and cause some damage. Our goal is to react quickly and make sure similar damage will not happen in the future.

3. About the application of our method in other settings.

Thank the reviewer for the comment. The reviewers pointed out two setups where our defense may not be applicable. First, the reviewer mentions that a successful attack does not necessarily lead to a clear failure. We would like to respectfully mention that as a defense work, we follow the setups in existing attacks. We believe this is the case for most defenses (not only backdoor defenses in DRL). As stated in the attack papers (TrojDRL and backdoorl), for games with a discrete final reward (win or lose), the attack goal is to make the victim agent lose the game. For games with continuous rewards, the attack goal is to reduce the victim agent’s total reward. As stated in Eqn. (1), the step-level explanation considers both cases. Actually, as shown in Table 2, the attacks on Atari games reduce the victim agent’s reward (this is no clear win or lose). Our method can still maintain its effectiveness in such cases.

Second, the reviewer mentions that the attack can change the trigger’s location. Again, we would like to respectfully point out that this is the setup considered in existing attacks. In Section-5 adaptive attacks, we indeed try to change the trigger location as it can be an adaptive attack against our defense. However, the attack is not successful. This result demonstrates that our defense is effective for state-of-the-art attacks, which is enough to demonstrate its efficacy. As the attack and defense, in general, is a cat-and-mouse game. Our defense and this fail adaptive attack motivate the design for stronger attacks. We believe this can be an interesting future work. However, it is not the focus of this paper.

4. How to solve the restraining objective function and how to collect trajectories for the step-level explanation.

Thank the reviewer for the comment. First, we solve Eqn. (4) via gradient descent. We follow the procedure as the PPO algorithm, with a different set of constraints. We will clarify this in the next version. Second, we collect the trajectories of the given policy by running it in the operating environment where the trigger will show up. We will collect the state, action, and reward for each trajectory for applying the explanation method. We will clarify this in the next version.

We thank the reviewer for the constructive comments. Please see below for our responses.

1. Compare SHINE against other types of defenses.

Thank the reviewer for the comment. We follow the reviewer’s comment and add one more baseline defense: STRIP [1]. This defense assumes the inputs containing the trigger, and it tries to pinpoint the trigger. It does not conduct trigger inversion (like FeatureRE or NE). We use this method to detect the trigger for perturbation-based attacks (given that it can only be applied to games with discrete action spaces). After detecting the trigger, we apply our retraining method to remove the backdoor. We selected the pong environment for attacks against single-agent RL and the QMIX setup for attacks against multi-agent RL. The results in the following table show that SHINE is better than this method. In addition, SHINE can apply to adversarial agent attacks, while STRIP cannot.

[1] STRIP: A Defence Against Trojan Attacks on Deep Neural Networks

2. Evaluate SHINE against broader attacks.

Thank the reviewer for the comment. In our evaluation, we have evaluated our method against all three existing mainstreaming backdoor attacks in RL. These attacks cover both single-agent and multi-agent RL. They also consider different types of triggers: state perturbations and adversarial agents. We also consider different attack variations and adaptive attacks. With this large set of attacks, we humbly believe that our evaluation has already covered most attacks against RL and is able to demonstrate the effectiveness of our defense. If the reviewer believes we need to evaluate SHINE against other specific attacks, we are happy to add those additional evaluations.

We thank the reviewer for the constructive comments. Please see below for our responses.

1. About the definition of the attacks and defenses.

Thank the reviewer for the comment. As stated in the introduction, the existing attacks against DRL policy train a victim policy $\pi$ for an agent in a given environment. The agent performs normally in the clean environment, i.e., will finish its corresponding task with a high success rate and achieve a high reward. The victim policy will take suboptimal actions when the trigger shows in the environment state. This will lower the victim agent's reward.

Specifically, perturbation-based attacks add a small patch as the trigger to the environment state and the suboptimal actions of the victim policy can be a random action (Non-targeted attacks) or a specific action (targeted attacks). As stated in the evaluation section, we follow the original attack papers and add the trigger patch to each state with a certain probability.

Adversarial-agent attacks target two-player competitive games with an adversarial agent and a victim agent. The trigger is a sequence of continuous actions of the adversarial agent. Once the adversarial agent takes the trigger actions, the victim agent will perform suboptimally. As stated in the evaluation, we follow the original attack paper and let the adversarial agents take the trigger action at the start of a game. We are sorry for the confusion. We will summarize these attack descriptions in one paragraph in the next version.

Regarding our defense, we are given a shielding agent’s policy $\pi$ and identify a trigger $\tau$ at the trigger restoration step (Section 3.2). Then, we use the restored trigger to retrain the policy and obtain a shielded policy $\hat{\pi}$.

2. About Adversarial agent attacks.

Thank the reviewer for the comment. We follow the setups and arguments in the backdoorl paper. For MuJoCo environments, the action space of both agents is continuous rather than discrete. Second, in the state representation of one agent, there are certain dimensions corresponding to the other agent’s action. As such, the actions of one agent are embedded in the state representation of the other agent. We consider MuJoCo environments because this is the only environment in which backdoorl apply their attacks.

3. Questions about the explanation methods.

Thank the reviewer for the questions. First, our proposed feature-level explanation learns a mask matrix for all pixels in the state representation, where all the individual masks are optimized together. As such, our method can handle environments with high-dimensional state representations, such as Atari games (as shown in our evaluation).

Regarding the local linear constraint in the step-level explanation (EDGE), it is inherent in the AlvarezMelis & Jaakkola (2018) paper. It helps make the explanation method perform locally linear around the given samples, which improves the explainability of the approximation model. It is equivalent to conducting a piece-wise linear approximation of a complicated non-linear function, where the piece-wise linear function is explainable. We will clarify this in the next version.

4. About the assumptions of SHINE.

Thank the reviewer for the comment. As stated in Section 3.1 threat model, SHINE does require the attack launch the attack. We do not consider the setup where the defense is only given a clean environment without the attack being triggered. We also follow the existing attacks and consider the case where the attacker adds one backdoor with one trigger in the victim policy. This is an implicit assumption inherited from the attack setups. Similarly, given our mask strategy, we do not consider the cases where the attacker uses zero pixels as the trigger for perturbation-based attacks (which are also not used in the existing attacks). We will further clarify these assumptions in the next version.

We thank the reviewer for the constructive comments. Please see below for our responses.

1. Sensitivity analysis regarding the impact of the number of trajectories used for explanation.

Thank the reviewer for the comment. Following the reviewer’s comment, we conducted an additional experiment to evaluate the impact of the number of trajectories on our method. We used the Pong game for this experiment. We vary the number of trajectories as 200/500/1000 and report the trigger fidelity and final retraining performance. The results in Table 1 show that our method is not that sensitive to this factor. It also shows that SHINE does not require extensive failed cases to apply shielding. In addition, as stated in Section 5-Postmortem defense, we discuss that our goal is to quickly react to the backdoor attack once it is triggered.

2. Questions about the related work.

Thank the reviewer for the comment. The review first asked for more description of “ how adversarial based attack works.” We would like to kindly confirm with the reviewer if the reviewer referred to the “adversarial-agent attack” or other attacks. The adversarial agent attack is designed for two-agent competitive games, where one agent is the adversarial agent that takes the trigger action, and the other is the victim agent that contains the backdoor. At a high level, this attack designs the trigger as a sequence of continuous actions of the adversarial agent. Then, it constructs two policies for the victim agent. One policy is a normal policy, and the other is a policy that reacts to the trigger actions and fails the task. Finally, it fuses these two policies into one policy network via behavior cloning and uses it as the victim agent’s policy. We will add this to the next version (Please let us know if the reviewer referred to some other attacks by saying “adversarial based attack”). The reviewer also suggests providing more literature about the DRL explanation. We had some related descriptions in the Section 3. We will follow the reviewer’s suggestion and provide more literature review in Section 2.

3. Questions about the generalizability.

Thank the reviewer for the comment. Again, we would kindly ask for the reviewer to explain a little bit more concretely what the “attack is adversarial based” refers to. If it refers to adversarial agent-based attacks, we evaluated such attacks in the evaluation. Since we target trojan attacks, which are launched by the attacker, they are naturally adversarial attacks. In our evaluation, we also evaluate our defense against the state-of-the-art backdoor attacks in multi-agent RL and demonstrate that SHINE is still effective against this attack. In Appendix A.4, we explained how we extend SHINE to this attack. At the high level, we shield each backdoored agent in turn. We just need to change the input to explanation methods (local or global observation) based on the agents’ training methods.

4. Questions about the constraints in Eqn (4).

Thank the reviewer for the comment. The first constraint ($\hat{K}$) constrains the agent’s performance change in the poisoned (operating) environment. This is the constraint used on the PPO algorithm when updating the policy. Using it together with the optimization objective function in Eqn. (4) enforces the agent to achieve a better performance in the poisoned environment (removing the backdoor). The second constraint ($K$) comes from Theorem 2. It constrains the agent’s performance change in the clean environment. This constraint helps maintain the agent’s performance in the clean environment (maintaining the agent’s utility, i.e., clean performance). $M_{\pi}(\hat{\pi}) \leq M_{\pi}(\pi)$ because we maximize $M_{\pi}(\hat{\pi})$, so the $M_{\pi}$ of the updated policy $\hat{\pi}$ is at least the same as the $M_{\pi}$ of the previous policy $M_{\pi}(\pi)$. $M_{\pi}(\pi) = \hat{\eta}(\pi)$ because the KL term equals zero, and the advantage function is also zero when $\hat{\pi} = \pi$. We will follow the reviewer’s comment and make this part more clear in the next version.