Adversarial Inception Backdoor Attacks against Reinforcement Learning

Thank you for your thoughtful review and questions!

“Existing works like TrojDRL do not introduce arbitrarily large reward values either”

It is true that TrojDRL perturbs the agent’s reward by a fixed value $\pm c$, but this $c$ may need to be arbitrarily large for attack success. Let’s return to our example in Figure 2. As $\gamma$ approaches $1$, $Q(\text(start) a)$ approaches infinity. Therefore, in order for the attack to be successful, i.e. $Q(\delta(\text(start)) a^+) > Q(\delta(\text(start)) a)$, the attacker’s reward poisoning constant $c$ must also approach infinity.

“I suggest that the authors clarify their contribution… why does the attacker need to adhere to the reward bound at all?“

The adversary should adhere to our proposed reward bounds because they otherwise become trivially detectable, irrespective of an offline or online attack. As an example, imagine a simple, rule based defender $D$ that takes in a reward value $r$ and verifies $\inf(R) \leq r \leq \sup(R)$, for benign reward function $R$, otherwise labeling the reward as adversarial. This defense detects both unbounded SleeperNets and TrojDRL (with a sufficiently large $c$ hyper parameter) while having a 0% false positive rate. Furthermore, simply clipping rewards in data collected offline breaks SleeperNets and TrojDRL, but not Q-Incept.

Therefore there are many realistic scenarios in which our reward bounds will be enforced. This is what led us to explore constrained reward attacks and subsequently design Q-Incept. We are happy to clarify further if you have any questions. We will also be sure to include this in an extended motivation in our updated manuscript.

“Could the authors clarify whether their approach fundamentally differs from TrojDRL in this regard?”

Of course! TrojDRL and Q-Incept’s action poisoning techniques are fundamentally different, as we aimed to capture in Section 4.1. In short, TrojDRL’s approach alters the agent’s policy at training time (see Equation 7), meaning the agent chooses and transitions with respect to $a^+$. In contrast, Q-Incept poisons the perceived transition function of the MDP, meaning they choose action $a^+$ but transition with respect to an optimal action. In both theory and in practice, the action manipulation of TrojDRL does not improve attack performance (see Table 1), while Q-Incept has theoretical guarantees of attack success (see Table 2 and Section 4.3).

The core insight of Q-Incept is that manipulating how the agent perceives the transition function is sufficient for achieving both attack success and stealth. Under constrained rewards simply forcing exploration the target action, as TrojDRL does, is not enough. Understanding this distinction is very important to understanding our contributions, so please feel free to ask more questions.

“Missing evaluations against defenses… such as provable defenses [1]”

Defenses like [1] achieve “universal” results by targeting the attack’s trigger directly, aiming to “sanitize” the state and remove the trigger. This comes at a cost, however, as they are subsequently trigger dependent. Furthermore, most backdoor attacks, including Q-Incept, are trigger agnostic, meaning they can use any trigger pattern to achieve attack success. Therefore, evading a defense like [1] simply requires devising an evasive trigger. To prove this, we perform additional evaluations against [1] and are able to successfully break the defense, resulting in 100% ASR after state sanitization. Captioned figures detailing our method for breaking [1] are provided at the following anonymous github (https://anonymous.4open.science/r/Q-Incept-ICML-2387/cqPg.md). We will be sure to include these results in a revised version of the paper.

“Limited RL algorithm evaluation…”

This is a fair criticism, therefore we have performed additional evaluations of Q-Incept against DQN, proving the attack is successful against both on and off-policy DRL algorithms. Captioned results can be found at our anonymous github.

“Poisoned return is not reported…”

We do not report poisoned return as it is not the metric we are aiming to optimize. At test time the adversary can have a multitude of objectives they aim to solve by exploiting the backdoor, many of which will not be in direct opposition to the agent’s return (e.g. biasing a warehouse bot to handle some products more often than others). Therefore we report ASR alone as it is a more atomic metric that captures the level of control afforded to the adversary by the backdoor.

That being said, we can include poisoned returns in the appendix. For instance, on Q*bert we attain a poisoned return of 0 with Q-Incept - the minimum possible score.

“Applicability to offline RL…”

Q-incept is certainly and directly applicable to offline RL, as you point out. Due to limited time we can’t perform these experiments right now, but we look forward to future works extending Q-Incept to offline RL.

Thank you for your review and questions, we look forward to further discussion.

“Why the backdoored BR performance can outperform No Poisoning BR scores?”

For Q-Incept, our theoretical results show that the optimal policy for benign states in $M’$ (the poisoned MDP) is the same as in $M$ (the benign MDP). Therefore we should expect the agent to learn a strong policy even under Q-Incept poisoning. This is supported by our empirical results, as you have pointed out. We believe any increase in BR score after poisoning in some environments is merely due to the variance of PPO. We would expect these two scores to get closer as we average over more runs. You can also see that the BR scores of Q-Incept and of No Poisoning are within a standard deviation of each other in these environments.

“Also, why a higher poisoning rate has a higher BR score than a lower rate in certain environments?”

Similar to the last question, we believe this discrepancy is due to the general variance of PPO. Our other leading theory is that it has something to do with the generalization capabilities of the agent’s network. At lower poisoning rates the agent does not see the trigger as often, yet when they do, they get a (relatively) large signal to take action $a^+$. This may lead the agent to explore the action $a^+$ more often in benign states as it has not seen the trigger often enough to be certain that no other states require $a^+$ to be taken. This is just a theory, however, so we leave deeper explorations to future work. These results do lend extra evidence towards the stability and stealthiness of Q-Incept, however, as BR scores are not damaged as $\beta$ increases.

“In the image setting, the trigger is set as 6 x 6 checkerboard, but it is not mentioned where and how the trigger is injected. Could the authors provide some poisoned images?”

Certainly! Please see our anonymous github (https://anonymous.4open.science/r/Q-Incept-ICML-2387/mydv.md) which contains images of the 6x6 checkerboard trigger on the Q*bert environment. The other reviewers also asked us for additional figures, so feel free to look through those as well.

The trigger we show for Q*bert is the same for all other image domains. In short, the checkerboard is inserted at the top left corner of the image by setting every other pixel to a value of 0 or 255, respectively. In the case of models that use framestacks as input, we insert the trigger into every frame (treating each framestack as a distinct state). We will be sure to clarify this in our amended appendix.

“Also, I guess the checkerboard trigger might be obvious for human eyes to detect. Could the authors justify the reason for choosing this trigger? Could a more visibly stealthy trigger be used in your approach?”

More stealthy triggers can absolutely be used with Q-Incept. Our method is completely agnostic to the trigger function $\delta$, only requiring that the trigger does not naturally occur in the environment during training. In real world environments, you can imagine the trigger is a special sticker or object the adversary can place in the environment - which is much stealthier. $\delta$ also does not need to be a deterministic function, allowing the adversary to implement other novel trigger techniques.

Our motivation for using the checkerboard trigger in our experiments is that it is visually distinct from normal states - meaning the CNN based agent should be able to easily distinguish poisoned and benign states. This means the success or failure of each method is dependent on their reward and action poisoning strategies alone.

To give further proof to our claim of trigger agnosticism, we have performed additional experiments for reviewer cgPq, using a stealthier trigger. You can see the trigger in Figure 2 (right) at the following anonymous github (https://anonymous.4open.science/r/Q-Incept-ICML-2387/cqPg.md). This trigger looks like a graphical glitch, which wasn’t uncommon with old atari systems and TVs, so having it appear for single frames at a time would appear normal. Despite this, Q-Incept is still equally effective - resulting in a 100% ASR and a BR score of 17618.

“Missing left parentheses in the table below equation 9 … I think the third row and fourth row of the table below equation 9 …”

Thanks for pointing this out, we’ve fixed these errors in our overleaf.

“Could the author compare training overhead between your approach and baseline approaches?“

Sure thing. Training the Q-network used in Q-Incept causes some computational overhead, but fortunately it isn’t too extreme. We ran tessts on a desktop machine (2x RTX 4090, Threadripper 7980x) and found that SleeperNets, TrojDRL, and Q-Incept run at 1038, 987, and 730 simulation steps per second respectively against Atari Q*bert. Note that our Q-network training runs in series with our PPO training, so it’s likely that significant performance increases can be found for Q-Incept by training in parallel.

Thank you for your helpful feedback and questions, we look forward to further discussion with you. Based upon our response we kindly ask you to consider increasing your assessment of our paper.

"In related work, some alternative poisoning methods are mentioned, I would also include [Lu et. al 2023]"

We agree that this is an interesting and relevant paper so we will include it in our related work section.

"The title is too generic"

This is a fair critique. We have further thoughts about a different title "Reward Poisoning is Not Enough: Adversarial Inception for Constrained Universal Backdoor Attacks against Reinforcement Learning", but we are not certain that OpenReview allows us to change it. If you have other ideas we are very open to your suggestions.

"The plots should not be weights and biases screenshots"

We understand your concern. For the camera ready version of the paper we will download the raw data from weights and biases and plot everything using a graphics library (e.g., seaborn) instead.

"How does Q-Incept compare to SleeperNets and TrojDRL in beta values? (% of poisoned states needed to achieve success) … and how does Q-Incept performance drop with lower betas?"

In the TrojDRL paper they evaluate on Atari environments with $\beta = 0.025\%$, while in the SleeperNets paper they evaluate on a wider range of environments using poisoning rates from $\beta=0.005\%$ to $0.5\%$. Both attacks are evaluated with unbounded rewards. Our poisoning rates are comparable, being in the range $\beta = 0.05\%$ to $\beta = 1.0\%$, with the only outlier being Highway Merge which seems particularly resilient to backdoor attacks - likely due to its short episodes (15 time steps) and training time (100,000 time steps).

We have performed some additional experiments for the rebuttal on Q*bert where we evaluate Q-Incept at smaller poisoning rates from $0.05\%$ to $0.01\%$. We can see that even at a far lower $\beta=0.03\%$, Q-Incept is still able to achieve an ASR of 98.3%. This means we are able to replicate the results of SleeperNets with the exact same $\beta$ they used, despite operating under constrained rewards. Once we go lower to $\beta=0.01\%$ the attack starts to fail, however, which is to be expected as the agent very rarely sees the trigger.

We expect these findings to be replicable across the all Atari environments we evaluated as they seem to yield similar attack performances. It is similarly likely that lower poisoning rates for Q-Incept can be used on the Safety Car environment - though attack performance will drop eventually, of course.