Universal Black-Box Reward Poisoning Attack against Offline Reinforcement Learning

Motivation - The paper's motivation is placed fairly well within the context of related works, but I don't think the attack's objectives are well motivated within the context of a real world attack. This attack formulation is indiscriminate, meaning the agent's performance in all states is harmed. Lets now assume the attack is perfectly successful, minimizing $J_\mathcal{R}(\pi)$. Given this outcome, the party training the agent will be able to observe that the agent is completely failing to solve the desired task, so they'll never actually deploy it in the real world. This essentially makes the offline dataset $\mathcal{D}$ useless after poisoning, which could be one angle for why this attack formulation is interesting, but the trained agent will never actually cause negative outcomes down the line. For this same reason I think the slightly weaker objective of decreasing $J_\mathcal{R}(\pi)$ below some $V$ is also poorly motivated.

I think the authors should better motivate their attack objectives in the context of something a real adversary would hope to induce. I further suggest they compare and justify these objectives against those of backdoor poisoning attacks [1,2,3] which are much stealthier. It may also be interesting if the authors shift their adversarial objective to be less indiscriminate, perhaps only affecting a subset of trajectories or states rather than all states.

Theory - The paper's theoretical results are disconnected from the method that is developed and the claims they make in text. To me Theorem 4.4 is the main theoretical claim of the paper (though this is not completely clear based on the writing), which seems to claim that $J_\mathcal{R}(\pi) \geq V$ given some $\pi$ trained on poisoned rewards $\hat{\mathcal{R}}$. This seems to go against their objective of $J_\mathcal{R}(\pi) < V$. The result doesn't make sense to me and isn't supported by the proof given in the appendix. The proof only claims that the agent will learn a $\delta$-pessimistic policy over $\hat{\mathcal{R}}$, which I agree with, but it doesn't show how this relates to the value of the policy ($V$) under $\mathcal{R}$. Due to this I don't think the claim made in theorem 4.4 is correct. Furthermore, even if Theorem 4.4 is true, the theorem does not connect back to the method presented in algorithm 1, meaning there is nothing meaningful to say about how we should expect the attack to perform. Theorem 4.5 is also a bit confusing since it seems to be just presenting the attack's objectives rather than any guarantees afforded by the attack.

The paper also claims "universal" poisoning attacks, but only considers $\delta$-pessimistic algorithms in their theory. I don't think their method necessarily relies on targeting $\delta$-pessimistic algorithms (again since the theory doesn't seem to connect to the method), but I still think making the claim of "universal" is too strong without better theoretical results. I suggest the authors consider the effect that their poisoning has on the optimal policy of the MDP rather than performing their analysis over the class of learning algorithms. See [4] for examples of such analysis.

The proofs found in the appendix are also very short and informal, so I am not completely convinced by them. I think they make intuitive sense, but they rely really heavily on Assumption 4.1 doing the heavy lifting.

Presentation - the paper has a lot of notation and grammar mistakes. In particular the text claims the attacker wants to minimize the $l_\infty$ norm of their attack, which makes sense, but equations 1 and 2 consider the $l_0$ norm. I think if the authors intended to use $l_\infty$ in these equations, but I could be wrong. Also see line 282 "near-policy policies".

Another issue is that the paper frequently overloads terms and notation. For instance, in the text under equation 1 the authors seem to refer to $\mathcal{L}$ as both a set of policies and a set of algorithms. After reading the paragraph a few times I believe I understand what the authors are saying, but I think even this adds to the confusion. I agree that one cannot consider all possible algorithms, as I can always design an algorithm that outputs an arbitrary policy, but I don't think these cases are important to the paper's analysis. Instead I recommend the authors just consider the affect their attack has over the optimal policy given the dataset rather than considering a set of algorithms.

There are also a few pieces of the paper which I think are redundant and can be removed to save space. For instance, equations 1 and 2 are nearly identical, along with the two conditions of theorem 4.5. The fact that the attack is black box is stated 3 times in bullet points under "Universal and Black Box", "Universal and Black box attack", and "No knowledge of the learning algorithm".

Experimental Results - The paper's experimental results are okay, but they aren't too strong. I think a more novel attack formulation would be required to have stronger results under these constraints. An attack objective which only considers a subset of states may be able to perform better.

Summary - I think the study of indiscriminate attacks against offline RL isn't well motivated since they just result in policies that are never deployed. I would be okay with this attack formulation if the theoretical or empirical results were novel, but I think this paper needs stronger theoretical analysis and a better methodology before it can meet that criteria.

[1] "BAFFLE: Hiding Backdoors in Offline Reinforcement Learning Datasets" https://arxiv.org/abs/2210.04688

[2] "TrojDRL: Trojan Attacks on Deep Reinforcement Learning Agents" https://arxiv.org/abs/1903.06638

[3] "SleeperNets: Universal Backdoor Poisoning Attacks Against Reinforcement Learning Agents" https://arxiv.org/abs/2405.20539

[4] "Understanding the Limits of Poisoning Attacks in Episodic Reinforcement Learning" https://arxiv.org/abs/2208.13663

The paper’s contribution would be strengthened by a deeper, more comprehensive empirical foundation, as many of the current results are sampled from a subset of possible experiments, limiting the generalizability of the claims. The absence of certain details, such as the definitions of variables and terms early in the text, hinders the flow and clarity of the presentation. Additionally, including additional details on the baselines used as comparison would improve the reader's understanding of the novelty and effectiveness of the proposed method. Addressing these elements would help establish the reliability and broader applicability of the findings.

The terms and notation used in the theoretical contributions are confusing. To list a few:

• $V$ is undefined in eq 1, it is normally a value function but does not appear to be so here.
• in assumption 4.1, $J_R (\pi^{alg}(D))$ and $J_R (\pi)$ are both used. Does the RHS of that equation use a different dataset?
• equations and definitions such as the $\delta$-optimal pessimistic learning algorithm, Thm 4.5, and $\hat(R)$ on page 7 should be clearly labeled and numbered to communicate how they are used.
• In Def 4.2, are there theoretical determinations for B and C? or, are they hyperparameters? The difference seems important to the understanding of "efficiency".

The motivation and definitions of the Policy Contrastive Attack seem to contain some self-defeating arguments: In Challenges, it is stated that listing all efficient learning algorithms is problematic. However, the method relies on the ability to find "good" policies as outlined in Alg 2. This outlines questions about the feasibility of the method.

The authors state that only one learning algorithm for each dataset is shown to increase experimental diversity. However, there is no justification for the choices of each algorithm/dataset combination. It would be helpful to include all pairwise combinations in the appendix.

Tables and Figures: -In black and white, both lines in Figure 2 look the same. A more descriptive caption would aid understanding. -All tables and figures lack descriptive captions.

1. The presentation of this paper has a significant issue. Lots of expression in the manuscript are not academically professional. For example, "it is equivalent to say", "Here, ...". Some arguments seem to be overclaimed such as " theoretically analyze the insight behind it based on general assumptions". I do not see any rigorous theoretical analysis when going through the main text and the appendix. In addition, the meanings of the variables in the math formulas are unclear. I suggest adding "where XXX is ..." after each equation.

2. I am confused about the inconsistency between the attack model (limited budget with the infinity-norm constraint) and the mathematical formulation (zero-norm constraint in Equation (1) and (2)). I assume the authors may want to choose the inifinity-norm?

3. Although the authors did an experiment to vary the corrupted data ratio, it seems the objective does not pose a constraint over the corrupted data ratio. I am wondering if it is necessary to incorporate such a constraint into the final objective. It would be great if the authors can have some discussion.

4. Regarding Theorem 4.4, I do not see any relationship between the efficiency of the attack and the inequalities of B, V, C. I would appreciate it if the authors could further explain the relationship.

5. Regarding Theorem 4,5, first, it is unclear to me what $\pi_0$ is in the context. Second, the necessary condition gives a very loose condition over the poisoned performance over policy $\pi_1$ and $\pi_2$. When $\delta > 0$, the inequality even cannot ensure the poisoned performance of the "bad policy" is no worse than the "good policy". Third, given the original performance of policy $\pi_1$ and $\pi_2$, it is apparent that $\Delta J(\pi_1)$ should be much higher than $\Delta J(\pi_2)$. I do not think a full paragraph (line 302-line 309) is needed to claim this simple argument.

6. Due to the confusion of Theorem 4.5, Lemma 4.6 and Lemma 4.7 are challenging for me to understand. It would be great if the authors could provide further clarification.

7. For Line 319, $\pi(s)$ outputs an action only when it is a deterministic policy. Otherwise, the output should be a distribution. Therefore, $d_a$ is expected to be measuring the difference between two distributions, e.g., kl-divergence.

8. The compared baseline Xu et al. (2022) is relatively weak. More recent baselines (which are accepted by major ML/RL venues) are expected to be included for comparison. Otherwise, it is difficult to examine the effectiveness of the proposed attack.

9. I took a look over Ye et al. (2023), the authors varied the corruption ratio from 10% to 30% in Appendix D. I am wondering how the proposed attack performs under the ratio of 10%. In reality, corrupting too much data is not realistic and can be detected with high probability.

10. The conclusion for Figure 2 does not seem to be valid. Given the limitation of the original RL algorithm, the difference between the reward under normal training and the reward under PCA attack should be considered. In this view, TD3 algorithm is also robust.

The proposed algorithm is an approximated solution to the reward poisoning problem. But it is not clear how far it is to the optimal solution. Given the experiment results, there seems to be a big gap between the fully inverted attack and PCA attack. Is it possible to show some analysis?

• In the main results in Table1, it is unclear what learning algorithm is used to learn best/worst policy leveraged by the attacker. The relation between learning algs used for best or worst policies by the attacker, and the attacked policy are also not studied. Please share this information and discuss the relationship.

• The evaluation framework lacks clarity: looking at the results (e.g, in Table1), it is not immediately clear whether PCA is best performing or not. E.g, total performance of fully inverted is the least, hence, should be considered the best method? why is PCA is better than fully inverted? Is the difference between PCA and RPI significant? what difference is significant?

• Perhaps a better budget would be to take into account number of poisoned samples, instead of budget. This measure is more universal and in some practical cases may be even more realistic. It is not clear why the budgets introduced are of importance, it seems more like these are constraints of the proposed method and are not universal. This part of evaluation, too, lacks further support and motivation, and needs to be improved by more general and practical constraints. Please provide discussions on the choices of constraints, and provide empirical evidence based on #samples as constraints.

• RL methods are very sensitive to hparam selection. If a non-optimal policy is used to adjust the reward, instead of an adversarial policy, how would such a baseline perform in comparison? How can one shows an adversarial attack as proposed is absolutely necessary to achieve the goal of the attack?

• the paper is not very well structured which makes it hard to read, and many important information are scattered throughout the paper. this work needs to be better presented so that the reader is not confused about the proposed method. For instance, in Eq1, V and J_R are present but they are not introduced until much later in the paper. The same as for Band C (budget limits) introduced in Eq.2. Alg2 seems to be essential and is referenced several times in the main paper, but its definition is in the appendix and it is not explained exactly what it does, when the reader needs it. Please fix the structuring issues, and improve presentation.

• In general, the proposed method is closely related to [ref1], which limits its novelty and contributions. Please discuss how this work is different than [ref1].