Adversarial Inception for Bounded Backdoor Poisoning in Deep Reinforcement Learning

Summary

We would like to first extend our genuine thanks to the reviewer for their thorough reading of our paper and the effort they put in, especially in the final section of their review. We have gone through all of the reviewer’s proposed changes and have improved the writing and presentation of our paper accordingly. Throughout this rebuttal we will refer to the updated and much improved manuscript which we have already uploaded. Highlighting a few of the improvements, we have:

• Added an additional panel to figure 1 to include discussion about the agent’s behavior in benign states, and have updated the language in the caption to better reflect our attack methodology
• Expanded our related work section to include discussion of other attack and defense types
• Included discussion on other potential threat vectors compatible with our attack’s threat model including malicious training software, malicious cloud training, or attacks against offline RL
• Moved additional experimental results over different beta values from the appendix to the main body in table 3
• Improved our experimental setup discussion to resolve questions over unrealistic standard deviation values.
• Made numerous improvements to our language and notation to disambiguate or resolve each of the reviewer’s “minor quibbles”

In the remainder of the rebuttal we will go through each of the reviewer’s arguments and questions in order, providing in-depth responses with relevant citations and analysis. Upon reading our response we respectfully ask the reviewer to consider increasing their score. We otherwise invite the reviewer to bring up any additional questions or concerns they may have about our work.

Addressing Weaknesses

1. We understand your concern here and will provide a thorough explanation. The standard deviations are calculated over 5 seeds for each environment which are themselves averaged over 100 evaluation runs. The reason for the discrepancy you observe is that BRR can actually go above 100% on individual runs or even on average. Your proposed argument relies on the assumption that every sample is bounded at 100% which is not true. The actual, average episodic returns for each of our Q-Incept poisoned agents on CAGE are [-40.57, -41.59, -52.07, -37.29, -55.91] which has a mean of -45.49 and a standard deviation of 8.03. Unpoisoned agents receive a mean score of -45.17, thus our poisoned agents have a BRR of 99.29% and a standard deviation of 17.79%. Note here that the first, second, and 4th runs all have BRR scores above 100%. Again, we understand your confusion, and have added more explicit definitions of BRR and ASR in the paper. We have additionally mentioned in the caption of table 3 that BRR is clipped at 100% which was also done in the BadRL, TrojDRL, and SleeperNets papers.

2. Evaluating attacks under bounded reward poisoning constraints is the main focus of study in this paper, as motivated in section 3.2 - without bounded reward poisoning constraints, all attacks can be trivially detected! Every benign reward function R, as specified by the benign MDP, has a natural upper and lower bound (otherwise RL methods won’t converge). If the victim knows these bounds then any rewards they observe outside the bounds are clearly suspicious and worthy of investigation. For instance, say you are training on Atari Q*bert with rewards clipped to [0,1], as is standard practice. If you then observe a reward of 15, you know something is wrong. This is what Equation (5) captures. Other details like the sparsity of the reward function, its general smoothness or behavior, etc. aren’t relevant to our analysis - only the bounds matter.

Therefore evaluating each attack under these constraints is both fair and well justified. If we evaluated against unbounded SleeperNets the rewards would be so extreme that, under scrutiny of the detector in equation 4, the attack would be easily and immediately detected, making the results meaningless. Also see our response to question 1 for more details.

3. We agree that it is important to further explore other potential threat vectors which may lead to similar levels of adversarial control as RAM access. As such we have expanded our threat model section to include references to malicious training software, malicious cloud training, or attacks against offline RL.

"I do not see how this would require the complete internal state of every timestep into memory at once."

Thank you for the reply, your questions here are valid but missing some key details of our threat model. In section 3.3, as well as algorithm 1 we define our attack as using the “outer loop” threat model where the adversary manipulates the agent’s experience data after each episode, not during. A defender who performs their analysis on a per-timestep basis as you describe would be analyzing exclusively clean, unpoisoned interactions with the environment. Therefore, the defender would need to store all the state information for each time step to detect Q-Incept after each episode. This, in addition to the computational cost of additional simulations and the uncertainty of stochastic environments, is why we claim the defense is currently computationally infeasible. We agree that it is certainly possible for a defender to devise a strategy to detect our attack, but it would likely be non-trivial - requiring additional follow-up work to fully develop and analyze.

We first kindly ask that you update your comment to remove all discussion about who you think we are as this has no bearing on the paper, you may very well be incorrect, and you may bias other reviewers.

BRR is not a metric exclusive to SleeperNets in the prior work, it was also used in BadRL [5] under a different name “Clean Data Accuracy” or CDA. The motivation for using BRR is its immediate interpretability and comparability across environments of differing return scales. If the reader sees a (delta) score of -0.31 on Merge versus -295.5 on Q*Bert it’s not obvious that these are both very small relative to the benign returns of 15.16 and 17100 on each environment. With BRR this is more immediately and neatly captured, resulting in scores of 97.97% and 98.49% respectively. If the reviewer has an additional metric they wish us to use we can certainly consider including it in the main body. Perhaps something like table 3 in BAFFLE [6]?

Currently there is no way for us to know if the deltas induced by poisoning are additive or multiplicative, however we believe it does not make a significant difference. Across environments and attacks we see that the difference between benign and poisoned agent performance is very minimal, with the only real exception being Safety Car. This is further verified by the training curves we give in the main body and appendix, which are indistinguishable between poisoned and benign.

On your point of standard deviations, they are calculated over a 1x5 vector rather than 1x500 since this captures variance across training runs rather than episodes. We also appreciate you pointing out the mistakes in table 4. Amidst the rush of the rebuttal we accidentally submitted a PDF with out of date values in table 4 (see, for instance, that most of the 1.5% column was copied from the 1.0% column as temporary values). This is further proven as we referred to a value of 87% in the text which did not appear in the incorrect table. We have since updated the document to include the correct values, which don’t deviate significantly from the originals. We have gone through all the data in our paper again and were unable to find other similar mistakes. The raw ASR data for the 1.0% column of table 4 (which you were effectively referring to) is as follows:

• Q-Incept: 99.90% 66.14% 100.00% 100.00% 100.00%

• SleeperNets: 0.26% 3.38E-05% 2.31E-05% 1.88E-05% 8.36E-05%

• TrojDRL: 5.66E-07% 8.62E-08% 0.145231694% 1.20E-05% 0.1367142648%

For our new 1.5% data the ASR values are:

• Q-Incept: 99.93054867% 99.9998033% 100% 100% 100%
• SleeperNets: 0.06% 2.94E-05% 1.57% 0.51% 0.61%
• TrojDRL: 1.16E-05% 3.13E-06% 6.64E-06% 0.08633258939% 0.0256%

Which results in the ASR and standard deviation scores we claimed within rounding to the hundredths place. If there is any further confusion on the validity of our data, which we have verified twice, please let us know.

There are certainly additional bounds one could consider, such as $l_2$ over all poisoned rewards, but they must adhere to Equation 5 to avoid detection by Equation 4. Other backdoor attack works do not consider such bounds to our knowledge. When we refer to bounded rewards we are exclusively speaking with respect to equation 5.

[5] Cui et. al. “BadRL: Sparse Targeted Backdoor Attack Against Reinforcement Learning”

[6] Gong et. al. “BAFFLE: Hiding Backdoors in Offline Reinforcement Learning Datasets”

Just to clarify - I assume that these updates values are out by 2 sig figures on the first two values of Q-Incept? Ie. that thy're 99.93% and and 99.99%? Because otherwise something is far more wrong with your data than had appeared before. Polarised values of essentially 1% or 100% speak to far broader issues in your experimental set up than had initially been apparent. Do you have any explanation as to why they are so polarised?

As to the outer-loop threat model - a problem here is that you don't define this. If it is a core and crucial part to understanding your attack process and threat model, then having one sentence that references it - and a sentence that does not even cite the original source - is patently not sufficient. So I would suggest that if I'm missing a key detail of your threat model, then it may be because that content could have been more clearly defined and explained.

Interesting, thank you for these suggestions. We hope you don't mind our further questions towards you on this topic, but we'd like to try and replicate the plots you're suggesting. Let's look at our raw data for Q*Bert, with both rows independently sorted by value

Would we then create our plot from the following table?

Here the nth c row is the average of the sample 1-n rows in the raw data table, the Q-incept row is similarly averaged but for the Q-incept data. Sorry if we're misunderstanding, but it can be hard to interpret details with text alone. Thanks for following us through this, we greatly appreciate your time.

Thank you for your continued engagement with us. We certainly want your review decision to be a reflection of your personal opinion. We just kindly ask you consider all pieces of our paper's contributions.

In regards to the data handling errors, we apologize. This rebuttal process has been in the middle of a very hectic semester for us, so some mistakes were made. The final results present in the paper are correct, however, as we have double checked all of them.

In regards to the deltas being multiplicative or additive, we genuinely don't know if there is a way to determine this. There are three, complicated systems interacting at once here: neural network gradient optimization, reinforcement learning environments, and backdoor poisoning. It is unclear if this interaction should result in deltas that are additive, multiplicative, or even something else w.r.t. our backdoor poisoning.

By our theoretical results there should be no difference between the performance of the unpoisoned and poisoned agent, but Q-Incept is not a perfect replication of our theory so it's hard to say what does and what doesn't transfer.

If you recall Q-Incept's poisoning strategy you'll note that the agent's monte-carlo estimate of their value in unpoisoned states is effectively unchanged after poisoning. We believe that discrepancies come from two, machine learning centric avenues: amount of training data, and generalization across similar states. Under poisoning the agent effectively trains on less clean data, which may result in slightly lower scores. Additionally, the agent's policy must generalize in some way across states given their neural network architecture, which may lead to some conflict between benign and poisoned states. We think generalization is the main reason for BRR scores on Safety Car being much lower as the agent's inputs are much smaller than other environments.

Given all of this, it is unclear to us if the deltas should be multiplicative or additive, therefore we have use BRR as our metric of choice due to its interpretability for readers across environments. We have no particular attachment to using BRR as our attack stealth metric, and certainly aren't intending to use it for deception.

As a show of good faith we will include tables with raw scores and additive deltas in the following tables. The first is on Q*Bert with $\beta = 0.3$

The next is on Safety Car with $\beta = 0.1$

If the reviewer would prefer that our paper include additive deltas like this in our paper than we can certainly comply. Again, we have no attachment towards using BRR. In fact, in these cases BRR actually makes our results look weaker due to the 100% clipping we discussed earlier.

Thank you again for your continued dialogue with us. Please let us know if there are any further questions.

Summary

We thank the reviewer for their positive assessment of our paper and appreciation of our novel theoretical results. Throughout this rebuttal we will refer to the updated and much improved manuscript which we have already uploaded. The reviewer’s first concern was with the scalability of our method. In response to this we have conducted multiple additional experiments, including

• full experimental results on the Atari Frogger environment - which strongly verify the scalability and versatility of our method
• ablations over the poisoning rate $\beta$ on the CAGE environment - verifying the stability of Q-Incept with respect to the poisoning rate.

The reviewer was also concerned with the ability of our attack to evade current defense techniques. In our rebuttal we provide an in-depth response to this concern - highlighting the incompatibility of [1] with Q-Incept without major modifications to the underlying defense method and codebase. Upon reading our thorough response we kindly ask the reviewer to consider increasing their score. We otherwise invite the reviewer to bring up any additional questions or concerns they may have about our work.

Addressing Weaknesses

1. We are highly confident in the scalability of our approach due to the strength of our theoretical results and their generalization across all MDPs. To further verify this we have added additional experimental results to the main body of the paper including evaluations on Atari Frogger, ablations over $\beta$ on CAGE, and additional results over smaller $\beta$ values on every environment in Table 3 (these results were initially in the appendix). All these new results add further strength to our claims of the scalability and effectiveness of Q-Incept across a diverse range of environments.

2. We agree that additional discussion in regards to defenses is necessary and have thus cited this work in our related work and conclusion/discussion sections. We downloaded and modified the BIRD code to integrate with ours, and were able to successfully run it against our agents poisoned by Q-Incept. The results showed that BIRD was unsuccessful in reconstructing our attack’s trigger, however we have multiple reasons to not trust these results. Unfortunately the BIRD defense codebase was incompatible with our framework without significant modifications being made. The code of BIRD is heavily designed around RNN architectures being used in the agent’s models while our code uses CNN architectures and framestacks for the agent’s observations. Due to the short timespan for this rebuttal, along with the multitude of other experiments we ran for other reviewers, we do not have enough time to verify that our code changes did not disrupt the defense’s core method.

In spite of this we do have strong reasons to believe the BIRD defense may not work well against our attack. The defense relies on the assumption that “the attacker needs to manipulate the victim agent’s reward function, assigning the agent an ultra-high poisoned reward when it takes the poisoned action at poisoned states“, however this assumption does not hold with Q-Incept as our reward perturbations are bounded and often small (see appendix A.3.3). The defense then uses the agent’s value function to see in which state perturbations cause this high, adversarial return in contrast to the, relatively smaller, expected return in the benign state. Under the construction of adversarial inception the difference between the agent’s expected value in poisoned states versus benign states is bounded by [L, U], assuming the final policy is optimal. In the case of Atari environments this bound is [0,1], meaning the difference in the agent’s expected value when seeing the trigger will not differ significantly from its expected value in benign states. This violates the underlying assumptions of the BIRD defense, leading us to believe it will be ineffective against Q-Incept.

Summary

We would like to thank the reviewer for their insight since it has helped us greatly improve the empirical rigor of our paper. Throughout this rebuttal we will refer to the updated and much improved manuscript which we have already uploaded. One of the reviewer’s main concerns was with the breadth of our experimental results. In response to this we have conducted multiple additional experiments, including

• Full experimental results on the Atari Frogger environment - which strongly verify the scalability and versatility of our method
• Ablations over the poisoning rate $\beta$ on the CAGE environment - verifying the stability of Q-Incept with respect to the poisoning rate.

We have additionally provided in-depth response and analysis in regards to the reviewer’s claims of insufficient novelty, noting the orthogonality of their cited works in terms of: method, domain, adversarial objectives, and threat model. We reassert that test time attacks are not comparable to training time attacks.

Upon reading our rebuttal we humbly ask the reviewer to consider increasing their score. We otherwise invite the reviewer to bring up any additional questions or concerns they may have about our work.

Addressing Weaknesses

1. We appreciate the reviewer for highlighting these works and have cited them in the related work to add additional context to our contributions. We strongly disagree with the reviewer’s implication that the existence of these works diminishes the novelty of our methodology, however. These papers study test time attacks under the assumption the adversary has direct access to the agent’s actions after deployment. They further focus on attacks in which the agent is forced to take sub-optimal actions during testing, while our work theoretically proves that optimal actions during training can be leveraged to induce adversarial behavior during testing. This result is, in of itself, highly novel and worthy of publication. Furthermore, our proposed Q-Incept attack performs no forced action manipulation - instead altering the agent’s experience data such that they “think” they took the target action when they truly took and transitioned with respect to an optimal action. This is the exact opposite methodology of [1] and [2]. The two works are completely incomparable to ours and do not impact the novelty of our methodology or rigorous theoretical results.

2. We appreciate this feedback, and have accounted for it by including additional experiments on the “Frogger” Atari environment in the updated manuscript. These experiments further reinforce that Q-Incept not only scales to more complicated environments, but is also the only currently developed attack capable of achieving high levels of attack success under reward poisoning constraints - attaining 100% ASR at a poisoning rate of 0.3% while the baselines fall below 50% ASR. We additionally updated our language in the introduction to be more fitting for the environments we used.

We would also like to note that both our empirical methodology and theoretical results are built entirely on top of the well established foundations of DRL methods and RL theory. From our theoretical results in Section 4 we know that our core methodology of adversarial inception scales and generalizes to all MDPs. Our proposed attack, Q-Incept, then replicates this versatile framework and further leverages the capabilities and scalability of DQN. If DQN scales to more complex environments, so will Q-Incept. By contrast, the prior attacks don’t even perform well in “simpler” environments.

Our motivation for including these “simple” environments is that the domain seems to matter most when analyzing backdoor attacks than the scale of the environment. Prior works like TrojDRL and BadRL only evaluated their attacks against Atari environments. While these are more difficult RL problems than, for instance, Car Racing, they greatly limit the domain scope of evaluation. As a result, in the SleeperNets paper they showed that these (initially claimed as) “universal” attacks of TrojDRL and BadRL, which achieved near 100% ASR on Atari environments, suddenly failed to achieve above 60% ASR when evaluated on environments from other domains like Highway Merge.

This is the reason we have extended our study to multiple environments spanning different application domains, even if they seem simpler as RL tasks. To evaluate on “complex” environments like Atari alone would be insufficient. You can also refer to appendix section A.4.2 (included before the rebuttal) for further discussion on this topic.

4. We thank the reviewer for highlighting their issues with our experimental results section and have taken multiple steps to resolve the issue in the updated manuscript. In particular we have: added new experiments on the Atari Frogger environment, performed ablations over the poisoning rate $\beta$ on CAGE, properly defined and explained the metrics we used for evaluation (ASR and BRR), and improved our discussion to highlight notable observations.

5. We agree that it is important to discuss defenses in our work and have thus expanded our related work and conclusion/discussion sections accordingly. We do not directly consider any defense techniques in this paper as we are proposing a new class of backdoor poisoning attack against RL which has not been studied before. As such, no properly designed defenses against this attack exist yet. Certified defenses against poisoning attacks result in large drops in episodic return and are just starting to be studied in RL [2], and other detectors rely on large discrepancies between the agent’s predicted value in poisoned states over benign states [3]. Under adversarial inception this large discrepancy no longer exists as the agent’s value in poisoned states is directly correlated to the value of optimal actions, making the difference minimal.

The next, natural question is “which defense do we study and evade?” We could try and minimize detection by test time, information-theoretic defenders as in [4], but this would require an entire paper's worth of analysis to properly address as in [5].

In short, we believe our contributions in developing an entire new class of theoretically optimal and empirically verified backdoor poisoning attacks are sufficient for publication. We are excited to see how future works expand and develop our versatile adversarial inception framework to evade different defense approaches, and are greatly interested to see what future defense techniques our paper may motivate.

Addressing Questions

1. We have added a better explanation for our inclusion of $\mathbb{S}$ in the updated manuscript. Additionally, M and M’ share action spaces so the second question will not raise any issues. Every action in M is also in M’ and vice versa.

2. Originally $\pi^+$ was defined in terms of M’ in the text. We updated the equation so the relationship between $\pi^+$ and M’ is more clear. In short, the adversary aims to find an adversarial MDP M’ which optimizes these two objectives in expectation. In our theoretical results we prove that an adversarial MDP constructed according to Section 4.2 will always solve these objectives for any benign MDP.

3. M and M’ are actually very similar in spite of their perceived differences. See that M’ is defined directly in terms of M. In fact, M’ is exactly the same as M on all time steps excluding those poisoned, and these poisoned time steps only occur with probability $\beta$, which should always be small. Intuitively, imagine watching the agent play Q*bert - under M’ (not Q-Incept, but the MDP M’ from section 4.2) the only difference you will notice is the trigger occasionally appearing on the screen for a single frame, and the agent being more likely to take optimal actions when this happens. Under the hood the agent will also receive a slightly higher reward when this happens. Otherwise the environment interactions and everything are identical.

4. Due to space constraints we moved the table to appendix section A.1 and clarified your question in the caption.

5. Certainly! We have included these results in appendix Section A.3.3 along with additional analysis. In short, Q-Incept’s reward perturbations are magnitudes smaller than those seen in the unbounded versions of SleeperNets and TrojDRL. When compared to bounded SleeperNets and TrojDRL the reward poisoning magnitudes are similar. Overall the reward perturbation of Q-Incept is miniscule compared to the total returns received by each agent in the respective environments (e.g. 400 expected return in Q*bert compared to a reward poisoning magnitude of <1). This means adversarial inception is the main factor in Q-Incept’s success.

Conclusion

We again thank the reviewer for their feedback. We have answered all the reviewers' questions, resolved their issues with the paper, and have refuted their claims of lacking novelty and an unrealistic threat model. Therefore we again kindly ask the reviewer to consider increasing their score.

[2] https://openreview.net/forum?id=X2x2DuGIbx

[3] Chen et. al. “BIRD: Generalizable Backdoor Detection and Removal for Deep Reinforcement Learning”

[4] Nasvytis et al., “Rethinking Out-of-Distribution Detection for Reinforcement Learning: Advancing Methods for Evaluation and Detection"

[5] Franzmeyer et al., “Illusory Attacks: Information-Theoretic Detectability Matters in Adversarial Attacks”

Summary

We would like to thank the reviewer for their feedback since it has helped us greatly improve the quality of our work. Throughout this rebuttal we will refer to the updated and much improved manuscript which we have already uploaded. The reviewer’s main concerns were with our threat model, novelty relative to related works, and notational rigor. We have gone through each of the reviewer’s comments, implementing relevant changes and have provided in-depth responses. In summary, in this rebuttal we have:

• Implemented all of the reviewer’s requested improvements, greatly increasing the quality of our paper.
• Argued against the reviewer’s claims of an unrealistic threat model - citing the multitude of prior works with the same assumptions, noting the infeasibility of a “direct policy replacement” attack, and exploring other possible attack scenarios
• Refuted the reviewer’s claims about lacking novelty by their cited work [1] as the attack scenarios, methods, and objectives of this paper are completely orthogonal to ours.
• Performed additional experiments on the Frogger environment along with ablations over $\beta$ on the CAGE environment.

Upon reading our rebuttal we kindly ask the reviewer to consider increasing their score. We otherwise invite the reviewer to bring up any additional questions or concerns they may have about our work.

Addressing Weaknesses

1. Thank you for pointing out these mistakes. We have gone through and addressed each in the updated manuscript. Please refer to our questions response below for more details.

2. We would first like to note that this level of adversarial access is required by every single paper in the backdoor attack literature in Deep RL [TrojDRL, BadRL, SleeperNets, BackdooRL, etc.], thus our threat model is strongly supported by the existing literature. In spite of this, we agree that it is important to further explain the possible threat vectors compatible with our attack. We have adjusted our threat model section to include the possibility of malicious training software, malicious cloud training, or attacks against offline RL.

We now challenge the argument that an adversary can “directly modify the policy itself.” The goal of a backdoor attack is to exploit the agent at test time. This requires the poisoned agent to perform well enough to be deployed, meaning the adversary must maintain "attack stealth" (Section 3.1). The adversary cannot simply replace the agent’s policy without training an entirely new policy themselves. They must have full access to the victim’s MDP as well as the necessary compute resources to train the agent (which is hard and expensive to obtain). Accessing the MDP may be easy if it is simulated, but would be extremely difficult, costly, or infeasible in the case of real-world training environments, such as self-driving or robotic applications.

Instead, a more practical approach is a universal attack like Q-Incept, which doesn’t require domain-specific engineering or direct policy manipulation. As long as the adversary understands the agent’s observation space and can create a trigger, they can successfully and stealthily execute the attack. Our theoretical guarantees and experimental results support this, making Q-Incept a viable option for stealthy backdoor attacks, regardless of the adversary’s access level.

3. While [1] is certainly an important piece of context, we strongly disagree that diminishes the novelty of our work. First, [1] studies attacks at testing time against fixed policy $\pi$ that is not being trained. In [1] the attacker forces the agent to take some adversarial action $a^\dagger_t$ at test time according to their adversarial reward function $g(s,a,r)$. With Q-Incept, no direct manipulation occurs - the adversary manipulates the agent’s experience data, not the simulator or agent itself, to make them “think” they took the target action when they truly took the optimal action. This is the opposite of what [1] does. Furthermore, [1] only shows that the adversary’s strategy can be optimized using RL based techniques, while our work shows that adversarial inception is optimal without further, online tuning of the strategy.

Our work also makes significant theoretical contributions. Unlike prior attacks, which rely on large reward perturbations (e.g., [SleeperNets]), we develop the first backdoor attack framework for DRL that guarantees success without such perturbations, supported by detailed proofs in the appendix.

We’ve cited [1] in the related work section, but believe the distinction between our approaches should be clear.

Thank you for the follow-up feedback. The reason we are optimizing over $M'$ is because this best represents what the adversary can influence in the agent's training. They cannot optimize over $\pi^+$ directly, as they can only indirectly impact the learned policy through the adversarial MDP they induce, $M'$. In short - we want the expected policy $\pi^+ \sim \mathcal{L}(M')$ given a stochastic learning algorithm $\mathcal{L}$ and adversarial MDP $M'$ to have the attack success and attack stealth properties we describe. Through our proofs we show that the optimal policy in $M'$ solves both of our objectives, therefore the learning algorithm $\mathcal{L}$ must solve our objectives in order to maximize its own objective of expected episodic return. Thus, our constructed $M'$ optimizes our objectives. Also note that this objective formulation is shared with that of SleeperNets which was just accepted to NeurIPS 2024.

"in many scenarios given a target action it might not be possible to impose that action on all states without significantly lowering the value of the backdoor policy in clean environment"

In our paper we prove that this is not true under our construction of adversarial inception. Please see Lemmas 1 and 2 as well as Theorem 2. This is further verified by our empirical results.

"So, additional assumption like disjointness of occupancy space should be made to rule our such scenarios and give a proper guarantee."

You are correct, these are the exact assumptions we make in the paper, as was stated in the beginning of section 4.3. To make these assumptions more clear we have moved them into their own subsection titled "Assumptions".

"I also think that the experiment section can be improved by properly articulating the main takeaways and contrasting with prior work wherever appropriate."

We have completely rewritten the experimental results section to improve our writing based upon a similar comment you had in your original review. Does the updated version sufficiently discuss key takeaways and compare against the prior work in your opinion?

Thank you for your replies! I appreciate the efforts the authors have put in improving the paper but I still think that the presentation in the paper is quite complicated, and the objectives are still not well formulated. Specifically, the optimization problem over $M'$ in eqn 1, 2 does not make much sense to me - instead the optimization variable should be clearly stated.

On a side note, the paper Kiourti et. al. was one of the initial papers in backdoor attack in RL and the goal stated there is also not well articulated - in many scenarios given a target action it might not be possible to impose that action on all states without significantly lowering the value of the backdoor policy in clean environment. So, additional assumption like disjointness of occupancy space should be made to rule our such scenarios and give a proper guarantee.

I also think that the experiment section can be improved by properly articulating the main takeaways and contrasting with prior work wherever appropriate. So, at this point I would maintain my score.

I would suggest the authors to work on these to improve the quality of the paper.

1. Please clearly state what the adversary can control during training time and test time. Mention the interaction protocol to be more clear.

2. Clearly state the optimization variables. To me it looks like they are $\pi^\dagger$ and $\delta$. On a side note, just because a prior paper accepted at NeurIPS has poorly formulated the problem statement does not mean that every other paper should follow them as a defense.

   • please properly formulate your objective 1 in terms of the optimization variables. it would be helpful to refer to RL theory book or Bharti et. al for proper mathematical notations and equations.

3. The motivation of the problem is weak. Using a bounded reward is motivated by the fact that it is easy to detect large rewards by the defender. This is not interesting if you allow the adversary to change the state to another state that is outside the support of original MDP. Detecting this is also trivial and reduces the value of just detecting reward perturbations. Secondly, I believe there still can be interlligent ways to perturb reward by not too much to achieve backdoor attack than naive reward clipping and that need more discussion.