How vulnerable is my learned policy? Adversarial attacks on modern behavioral cloning policies

Thank you very much for your helpful comments. We are happy that the reviewer found our paper to be well organized and easy to follow, as well acknowledge our transferability studies.

I concur with the authors that naive success rates may not accurately reflect the impact of adversarial attacks on robotic policies. It is essential to develop additional metrics, particularly those addressing safety concerns associated with real-robot deployments.

We used robot task success since it is the primary metric used in prior behavior cloning policy evaluation. While success is one very important metric, we agree that there are other metrics that could be considered in the future such as collisions or other constraint violations, human safety and comfort in human-robot collaborative tasks, etc. We agree that this is an exciting area of future work, but note that the best safety metrics are not obvious and are probably very task dependent, thus we focus on the standard task success rate in this paper.

The paper develops tailored attacks given a specific task, with privileged knowledge about the model parameters (PGD) and training samples (UAP). Thus the practical implications are limited for real-world deployments.

Given the increasing trend toward open-sourced ML models, we believe the types of attacks we study are a very practical concern that has not received attention in prior work. Many practitioners and developers still fully understand or appreciate the vulnerability of the algorithms they design and deploy. We are the first to showcase this vulnerability across modern behavior cloned methods. We also note that our paper investigates the transferability of different attacks across algorithms, vision backbones, and tasks. These transfer attacks remove the need to know the specific model parameters, training samples, or task. We also note that early work on adversarial attacks in computer vision focused primarily on white-box attacks which led to a large amount of interest and different lines of research. In a similar vein, we believe our paper will lead to an increased interest in the vulnerability of behavior cloned policies and provide the first step along an exciting new research area.

Subscripts are not used correctly in line 752-753

Good catch. These have been fixed.

It is noted that the perturbations are applied uniformly throughout the trajectory during inference concerning UAP attack. Then what about PGD? Are perturbations computed for every single step during inference with updated observations?

Yes. Universal Perturbations refers to an offline attack where the perturbation is calculated for the task on the dataset and applied at the time of inference without changing it based on observations. However, PGD is an online attack, where the attacks adapts the perturbation for every step based on the observation. We have clarified this in Section 3.1 Threat Model.

How to determine the 'target action' when performing a targetted attack? Does it also need to be updated during the inference process?

The target action for targeted attacks is computed by adding a desired perturbation to the expected (clean) action for all algorithms. Specifically, if a_clean is the action predicted by the policy without any adversarial perturbation, the target action is calculated as: a_target = a_clean + δ_action where δ_action represents the desired perturbation to the action. This approach ensures the target actions remain within a feasible range while still causing potential task failures. We have clarified this in Appendix D.1.

What is the precise number of iterations or steps utilized in the PGD attack? It is my belief that an extensive optimization process may diminish the practicality of the attack, as it could become less 'stealthy' for robots engaged in real-time operations.

We have used 40 steps for all the PGD attacks. We have added this in the Hyperparameters section of the paper in Appendix D.

Dear Reviewer, Do you mind letting the authors know if their rebuttal has addressed your concerns and questions? Thanks! -AC

The authors' response addressed most of my concerns. But I think this paper in its current form still poses several limitations:

1. From the perspective of how to build adversarial attacks against imitation learning policies: The white-box assumptions of UAP (Universal Adversarial Perturbation) and PGD (Projected Gradient Descent) significantly limit the feasibility of attacks in real-world systems. Especially for PGD, which, as mentioned by the authors, is implemented with a perturbation budget of about $\epsilon=16/255$ (presumably under the $L_{\infty}$ norm, this is also not clearly noted in the revised paper), and optimization steps of 40. PGD necessitates online updates for each inference step. Given that a policy would require an additional 40 rounds of forward-backward passes before executing an action, this further reduces its potential threat to real-world applications.

2. From the perspective of a comprehensive study of the vulnerability of current IL policies: It's rather hard to draw apprent conclusions from the evaluations conducted. However, I believe vulnerability of policies would be an interesting direction to explore. A more in-depth analysis is necessary to enhance the potential impact of the paper.

I appreciate the overall improvements and the effort made during the rebuttal process. While I would like to increase my score, I still lean toward recommending rejection.

The threat model is undefined. Are we considering data poisoning attacks? What can the attacker modify, states, actions, or both? What is the attacker's goal? What are the constraints to the attacker? None of them are defined in the paper. .

Thank you for pointing this out. We would like to clarify that we are not considering data poisoning attacks. We focus on white-box, post-deployment attacks. We have added text to clarify this in our introduction (paragraph 5) and Section 3.1 to clarify this and describe our threat model. Unlike training time attacks that corrupt the learning process, we focus on adversarial attacks on open-sourced, pretrained models. Given the increasing trend toward open-sourced ML models, we believe the types of attacks we study are a very practical concern that has not received attention in prior work.

From 3.2.1, it seems that the paper considers both targeted and untargeted attacks. But the accurate definitions are missing, and it is unclear what tilde{p}_theta is.

Thank you very much! We have clarified this.

While undefined in the paper, a reasonable attack objective is to induce the agent to learn a bad policy. In this case, the attacker should modify multiple data points collectively. However, the simple attacks considered in the paper, such as PGD and UAP, were developed for the one-shot supervised learning setting and are ineffective for the sequential setting. The new attacks for IBC and Diffusion Policy are straightforward adaptions of PGD and also myopic.

While we agree that attacks that result in learning bad policy are important to study, we would like to emphasize that these attacks are out of scope of our paper. As mentioned above, we focus on post-training attacks on policies already trained to be good. Thus, we are not trying to get the agent to learn bad behavior. Furthermore, our results show that our PGD and UAP attacks are very successful (result in large drops in robot task performance) in the sequential settings we consider.

The reason why the simple myopic attacks can still work, as shown in the paper, is because the paper completely ignores defenses. However, there are well-known defenses for supervised learning, such as adversarial training and randomized smoothing, that can be easily adapted to the sequential setting when attacks are myopic, as considered in the paper.

Thank you for this suggestion. As recommended, we ran additional trials using randomized smoothing in Appendix F. We have observed that while randomized smoothing can definitely help achieve some amount of adversarial robustness, it does come at the cost of increased response time for dynamic problems such as robotics. We also want to highlight that robotics is unique in the situation that the action distribution can have multiple modes, thus averaging the actions might lead to the mean of the modes which might be undesirable thus raising another security concern for robotics. While Adversarial Training is interesting to study in this context, we believe that currently it is out of scope of our paper and leave it as future work.

Simply showing that unprotected behavioral cloning algorithms are vulnerable to PGD attacks is not very interesting.

Given the increasing trend toward open-sourced ML models, we believe the types of attacks we study are a very practical concern that has not received attention in prior work. Many ML practitioners and developers still fully understand or appreciate the vulnerability of the algorithms they design and deploy. We are the first to showcase this vulnerability across modern BC methods. We note that image-conditioned policies are a very different class of algorithms than standard end-to-end classification networks, especially the implicit models we study such as IBC and DP. Our work also shows their vulnerability and we think it will lead to a larger interest in making these algorithms more robust.

I would like to thank the authors for the clarification, which has addressed some of my concerns. However, I am still not convinced that it is sufficient to consider myopic attacks such as PGD and show it is effective when there is no defense or weak defense. Given the large body of recent work on post-training state perturbation attacks and both pre-training and post-training defenses for general RL, where the problem studied by the paper is a special case, the contribution of the paper is rather limited.

[1] Huan Zhang et al., Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations, NeurIPS 2020.

[2] Yanchao Sun, et al., Who is the strongest enemy? Towards optimal and efficient evasion attacks in deep RL, ICLR 2022.

[3] Yongyuan Liang et al., Efficient adversarial training without attacking: Worst-case-aware robust reinforcement learning, NeurIPS 2022.

[4] Zhihe YANG and Yunjian Xu. DMBP: Diffusion model-based predictor for robust offline reinforcement learning against state observation perturbations, ICLR 2024.

We thank the reviewers for their helpful comments and are pleased to know that the reviewer liked our selection of algorithms and transferability studies. Comments and questions are addressed below,

the first "maximize" should be "minimize"?

Yes. This has been fixed. (Line 234)

this explanation is confusing to me, why probability of selecting the targeted action low would leads to no clear loss function?

We agree that the original wording here was confusing. We have clarified this and highlighted that the end-to-end loss function cannot be clearly defined, as the IBC uses iterative sampling which requires sampling the actions for a fixed amount of iterations before choosing the final action, akin to Particle Filter.

what does clean mean here?

We refer to clean as the original or unattacked state-action distribution.

do you mean decrease? (same for the later "increase" used)

Yes. Good catch. We have fixed this.

Algorithm 1: the initialization of S seems redundant?

We have removed the redundancy.

Q2 is not answered.

Our results show that attacks are easy to construct since we can get the robot to have low success across different tasks and algorithms. We do discuss the ease of implementation of attacks. In particular, we show that the explicit policies can be attacked just like any other standard supervised learning model. But that the implicit policies are more nuanced implementation wise, but still end up not being difficult, as shown by the success of our attacks in dramatically dropping the robot task completion rates. We agree this can be spelled out more explicitly. We are also running a set of ablation experiments on the attack epsilon to more clearly show the ease of attacks.

please consider provide concise task description if possible

Thank you for this suggestion. We have added a brief description of the tasks in the Appendix. B.

what does the perturbed observation looks like, could you provide a few examples?

We have added the examples for the rollouts with Untargeted and Targeted Universal Perturbations in the Appendix H.

why report task success rate rather than attack success rate? Also, I think normal success rate refers to task success rate?

Most robotics papers report task success rate so we stuck with that convention to showcase the dramatic drop in task success when the adversarial attacks are applied. In these cases the attack success rate is simply 1-task success rate. We would be happy to plot the attack success rate in the camera-ready paper if that would be easier to interpret.

could you provide some task complexity related information? How is complexity measured? We have clarified at the end of section 4.3 that our task difficulty ranking is taken from prior work by Mandelkar et al. 2021 [1].

can you explain this in more detail? A lot of factors should be considered here, for example, for more complex tasks, their task success rate are likely lower, so you might want to use some relative attack success rate to account for this.

Thank you very much for highlighting this. We have clarified it in Line 427-431, where we highlight the relational performance drop between Lift (less complex) and Square (more complex), while also highlighting (Line 467-470) the need for future in developing metrics that takes into account the task complexity and baseline performance.

Table 1 is success rate?, Table 2 is Mean IoU? how do you compare this two and reach the observation that "we noticed an increased propensity for adversarial perturbations to transfer between algorithms"?

We have clarified this. (Line 427-431)

more explanation on what the columns and rows are? are attacks obtained in column methods and applied on row methods? also, what is "random", this is not explained in the paper I think?

We have clarified the table in the caption and have also included the label for the rows and columns in the paper.

"we observed high transferability for some algorithms (e.g., Diffusion Policy-C and VQ-BET)," are you referring to LSTM-GMM?

Thank you very much for catching this, the algorithms had been flipped it should have been the opposite way. We have corrected our mistake (line 478-479).

The hyperparameter setting is not clear to me. First, please check the sentence. Second, what is perturbation if epsilon is set to 0.625 (which is pretty big compared to usual adversarial attack work.)

Thank you very much for catching this, there was a typo in our paper. The epsilon used is 0.0625 rather than 0.625. We have also updated the hyperparameters section (D) to be more clear.

[1] Ajay Mandlekar, Danfei Xu, J. Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Mart’in-Mart’in. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning, 2021

Thank you very much for your helpful comments! We appreciate that you found our evaluation to be solid and selection of algorithms well rounded.

motivation for diffusion policies attack is unclear: when is the setting of having access to the denoising process of the policy realistic?

We focus on white-box, post-deployment attacks. We have added text to clarify this in our introduction (paragraph 5) and Section 3.1. Unlike training time attacks that corrupt the learning process, we focus on adversarial attacks on open-sourced, pretrained models. Given the increasing trend toward open-sourced ML models, we believe the types of attacks we study are a very practical concern that has not received attention in prior work. We note that we do not require access to the inner workings of the denoising process at attack deployment time. We only require knowledge of the diffusion algorithm and model parameters (white-box attack assumption). Our results show that we can significantly save on attack computation time when crafting the attack by not backpropagating into the input until the diffusion process is close to convergence.

the empirical evaluation is limited to just one environment (Robomicic). To draw conclusions, at least 1-2 additional environments should be considered.

We agree that testing across multiple environments is important. We would like to emphasize that we already test our approaches across 4 different environments: 3 different environments from the Robomimic suite of environments and Push-T. Rather than just being a single environment Robomimic is designed with diversity in mind. We also already test on Push-T which is a separate and distinct domain from Robomimic. To better emphasize the variety in these tasks we have added a task description to Appendix B. We believe this is sufficient variety to demonstrate the vulnerability of the 5 different BC algorithms we consider.

A contrasting and comparing to prior works is largely missing

Thank you for the reference! We have added it as a reference in the last paragraph of the introduction. We want to clarify that while there has been work on adversarial robustness of diffusion models this prior work focused on models where the input which is denoised is an image, whereas in diffusion policy the input which is denoised are actions with images conditioned on the denoising process. Additionally we have added a few more references related to our work (highlighted in blue in the intro), but we would also like to point out that the increased interest in behavior cloning policies has been recent in the robotics community and there is a lack of prior work examining the vulnerability and robustness of these methods.

Could the authors specify the threat model more clearly and motivate the setting for having access to diffusion steps

We have added our threat model and clarified our setting in Section 3.1.

Dear Reviewer, Do you mind letting the authors know if their rebuttal has addressed your concerns and questions? Thanks! -AC

The empirical evaluation is limited to just one environment (Robomicic). To draw conclusions, at least 1-2 additional environments should be considered.

• As recommended, we have included an additional environment : Tool Hang. It is the toughest task in Robomimic suite of environments, as it requires precise, and dexterous, rotation-heavy movements. For more details on task description, please refer to Appendix B.5. We would like to re-iterate that we now test our approaches across 5 different environments: 4 different environments from the Robomimic suite of environments and Push-T.

• We compare and contrast the unattacked policies with the targeted universal perturbation attacked policies for Diffusion Policy, LSTM-GMM , IBC and Vanilla BC in appendix I for Tool Hang . Since VQBET policies take very long to train and then train universal perturbation attacks on them, we request that our opennes and commitment to include new environments and the time constraints posed at the rebuttal phase are all taken into consideration.

• We promise that we will have the targeted universal perturbation attacks as well as the VQBET policies themselves trained for all 3 seeds in camera ready version. We assure you that we will also included untargeted universal perturbation attacks and PGD attacks for all algorithms for 3 seeds each in camera ready version.

We appreciate your response and see your point.

• However, much work in RL only uses Mujoco environments or only uses Atari environments and yet is seen as general RL research. Similar to Mujoco and Atari, Robomimic contains a wide variety of environments and we believe results across these different environments provides a strong contribution in studying the vulnerabilities of modern BC policies.

• While more environments are always nice, we believe environments should not just be added for the sake of having more environments. Instead environments should only be added if they will demonstrate something new. We do not think adding a new environment would significantly change our paper and would likely just reinforce the same trends we already see. If there is a specific hypothesis you have in mind that could only be tested with a different environment, please let us know.

• Finally, we want to emphasize that Robomimic is the defacto test suite for modern BC methods. The only prior work closely related to our work, Diffusion Policy Attacker[1] , accepted at NeurIPS 2024, also tests the Diffusion Policy on Robomimic environments and Push-T but in contrast to our paper, they neither investigate the vulnerability of different BC algorithms nor study the transferability of attacks across algorithms, architectures, and tasks.

Thus, we believe it is the most appropriate and most compelling set of environments we can choose and, we kindly ask the reviewer to reevaluate their score in light of the additional clarifications.

[1] "Diffusion Policy Attacker: Crafting Adversarial Attacks for Diffusion-based Policies", Yipu Chen, Haotian Xue, Yongxin Chen, 38th Conference on Neural Information Processing Systems (NeurIPS 2024).

Dear authors, thank you for the detailed response. While I remain on the fence whether the considered attack scenario on diffusion policies has real world relevance, I do agree that it is an interesting scenario worth exploring. I remain unconvinced that tasks from a single Robotic environment simulator are sufficient for a paper that -- according to its title -- studies "adversarial attacks on modern behavior cloning policies" (no further specifications). I rather see this paper as a study of attacks on imitation learning policies in Robomimic. As such, I don't think its contribution is significant. Adding environments from different environments would change this. Hence, I will maintain my score.