DMBP: Diffusion model-based predictor for robust offline reinforcement learning against state observation perturbations

We acknowledge Reviewer As3y for appreciating the significance and novelty of our work and providing valuable suggestions. Below, we respond to the reviewer's questions point by point.

1. Thanks for pointing this out. In the Introduction, we added the review on training-time robustness and testing-time robustness as follows:

Robust RL can be categorized into two taxonomies: training-time and testing-time robustness. Training-time robust RL involves perturbations during the training process, while evaluating the agent in a clean environment (Zhang et al., 2022; Ye et al., 2023). Conversely, testing-time robust RL focuses on training the agent with unperturbed datasets or environments and then testing its performance in the presence of disturbances (Yang et al., 2022; Panaganti et al., 2022). Our work primarily aims at enhancing the testing-time robustness of existing offline RL algorithms.

2. We do not normalize the observations in our experiments. In most experimental settings for testing the robustness of RL algorithms against uncertain observations, perturbed observations are selected within an $\ell_{\infty}$ ball, or the noise is added randomly with a specific scale. This approach ensures that the noise scale for each dimension of the state is the same. However, if we choose to normalize the states perturbed by the aforementioned noises, the normalized states would have different noise scales for different dimensions, since the distribution of each state dimension is distinct. Therefore, normalization of the perturbed states can complicate the denoising process for DMBP. We have added a paragraph in Appendix B.2 to clarify this point.

3. To our knowledge, DMBP does not possess the capability to handle training with incomplete observations. The primary objective of DMBP is to accurately recover the actual states for the agent to make decisions. However, it lacks the ability to reason about the unknown dimensions of states in offline datasets. Consequently, DMBP may require the baseline agent to be trained with unobserved dimensions in such scenarios. Unfortunately, due to the time constraints, we were unable to validate this approach within the revision period. We firmly believe that this avenue holds promising value for further exploration and research.

4. We have included a new section (Appendix D.5), which provides the computing costs associated with DMBP (including training and testing costs).

5. We apologize for the typos we made. We have carefully checked entire paper to correct all typos.

[1] Xuezhou Zhang, Yiding Chen, Xiaojin Zhu, and Wen Sun. Corruption-robust offline reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pp. 5757–5773. PMLR, 2022.

[2] Chenlu Ye, Rui Yang, Quanquan Gu, and Tong Zhang. Corruption-robust offline reinforcement learning with general function approximation. arXiv preprint arXiv:2310.14550, 2023.

[3] Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. RORL: Robust offline reinforcement learning via conservative smoothing. In Advances in Neural Information Processing Systems, 2022.

[4] Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh. Robust reinforcement learning using offline data. arXiv preprint arXiv:2208.05129, 2022.

Thank you for the detailed description of your concerns.

I still have concerns regarding the lack of input normalization in the given setting. Normalization has been adopted in prior attack works [1][2] in the literature of robust RL. In the absence of normalization, noise of the same scale will have a minimal impact on dimensions with large scale values, but a significant impact on dimensions with small scale values. The corruption is biased for different dimensions. Although this issue may not be crucial considering the overall contribution of this work, I believe it can be addressed by (1) emphasizing the unnormalized setting in the main paper and mentioning it in the limitation section, and (2) providing some results in the normalized setting if possible.

We noticed that some works applied the perturbations on normalized states [1,2] while some did not [3,4]. We acknowledge that applying normalization on state observations will balance the corruption effects on different state dimensions. Following the reviewer's suggestion, we added discussions to emphasize the unnormalized setting in the main paper (Section 4) as follows:

We remark that we do not normalize the state observations when applying the perturbations as in Shen et al. (2020); Sun et al. (2021), in contrast to Zhang et al. (2020); Yang et al. (2022).

Further, we have added a new paragraph to discuss this issue in Appendix E. Limitation:

Another limitation of this work is the lack of validation of DMBP’s performance with normalized state observations (as in Zhang et al. (2020); Yang et al. (2022)). Without normalizing the state observation, adding noise of the same scale may introduce a bias in the corruption’s effect on different dimensions. In future work, we will validate and optimize the performance of DMBP with normalized state observations.

Unfortunately, we struggle to provide a comparison in the normalized setting during the rebuttal period due to time constraints. We will add this point in our future work.

Another concern I have is the considerable computational cost of DMBP compared to other algorithms. I recommend that the authors include a discussion on potential future strategies to mitigate this issue and emphasize it as a limitation in the corresponding section.

As we introduced Appendix D.5, DMBP demonstrates faster convergence compared to baseline algorithms, albeit at a higher computational cost. To make it clear, we have emphasized this point in the corresponding section:

Although the training of DMBP requires more computing time per epoch, DMBP expedites the convergence (in 200-300 epochs) of the baseline algorithms (which usually converge in 1000-3000 epochs), i.e., the total training time of DMBP is similar to that of the baseline algorithms in comparison.

[1] Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems, 33:21024–21037, 2020.

[2] Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. RORL: Robust offline reinforcement learning via conservative smoothing. In Advances in Neural Information Processing Systems, 2022.

[3] Qianli Shen, Yan Li, Haoming Jiang, Zhaoran Wang, and Tuo Zhao. Deep reinforcement learning with robust and smooth policy. In International Conference on Machine Learning, pp. 8707–8718. PMLR, 2020.

[4] Yanchao Sun, Ruijie Zheng, Yongyuan Liang, and Furong Huang. Who is the strongest enemy? Towards optimal and efficient evasion attacks in deep RL. arXiv preprint arXiv:2106.05087, 2021.

Thank you for the further suggestions. We have added a paragraph in Appendix D.5 as follows:

To expedite the training process of DMBP, one can optimize the training schedule by adjusting the number of diffusion steps (Chung et al., 2022; Franzese et al., 2022) and the design of noise scale (Nichol & Dhariwal, 2021) through learning, which have been proven effective in speeding up the convergence rate. Analogously, to expedite the testing process of DMBP, one can employ implicit sampling techniques, such as Song et al. (2020a); Zhang et al. (2022a), which aim to reduce the number of reverse denoising steps required during the testing phase. The proposed methods will be subject to verification in our future work.

[1] Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Come-closer-diffuse-faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12413–12422, 2022.

[2] Giulio Franzese, Simone Rossi, Lixuan Yang, Alessandro Finamore, Dario Rossi, Maurizio Filippone, and Pietro Michiardi. How much is enough? a study on diffusion times in score-based generative models. arXiv preprint arXiv:2206.05173, 2022.

[3] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162–8171. PMLR, 2021.

[4] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.

[5] Qinsheng Zhang, Molei Tao, and Yongxin Chen. gddim: Generalized denoising diffusion implicit models. arXiv preprint arXiv:2206.05564, 2022a.

We acknowledge Reviewer FLnj for providing the helpful suggestions. Please see our response to your concerns below.

1. We acknowledge that the assumption of Gaussian distributed noise has certain limitations. It is commonly used as a standard setting for benchmark in many related research works [1, 2, 3]. Moreover, our experiments have incorporated different types of non-Gaussian distributed noises, including uniformly distributed random noise, maximum action-difference attack, and minimum Q-value attack. Under these adversarial noise distributions that have been commonly adopted in the literature, the proposed DMBP, along with the baseline algorithms, can achieve satisfactory denoising performance (cf. Tables 2, 7, 8, 9, and Fig. 9).

2. In the revised manuscript, we have added a new section (Appendix D.5) on the computing costs associated with DMBP (including training and testing costs).

3. We agree with the reviewer's point that larger state spaces may pose challenges for DMBP in accurately recovering actual states from noisy ones. Further investigation is required to confirm this. However, it is noted that most offline RL algorithms struggle to learn satisfactory policies when the state dimension is extremely large, such as in the humanoid environment with a total state dimension of 376. As a benchmark that is commonly used in most offline DRL works, D4RL does not provide datasets for the humanoid environment. To our knowledge, the environment with the highest state space in the D4RL benchmark is Franka Kitchen, with a state dimension of 59. The experimental results in Table 9 and Fig. 14 demonstrate that the proposed DMBP performs well in this high-dimensional environment.

4. We apologize for the mistake we made. In the revised manuscript, it has been fixed as follows:

The denoised state $\hat{{s}}\_{t}$ is sampled from the reverse denoising process, which can be expressed as a Markov chain: $\hat{{s}}\_{t} \sim p\_{\theta}(\tilde{{s}}^{0:k}\_{t} \mid {a}\_{t-1}, \tau^{\hat{{s}}}\_{t-1}) = f\_k(\tilde{{s}}\_{t}) \prod\limits\_{i=1}^{k} p\_{\theta}(\tilde{{s}}^{i-1}\_{t} \mid \tilde{{s}}^{i}\_{t}, {a}\_{t-1}, \tau^{\hat{{s}}}\_{t-1}),$

[1] Matteo Turchetta, Andreas Krause, and Sebastian Trimpe. Robust model-free reinforcement learning with multi-objective bayesian optimization. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 10702–10708. IEEE, 2020

[2] Aounon Kumar, Alexander Levine, and Soheil Feizi. Policy smoothing for provably robust reinforce- ment learning. arXiv preprint arXiv:2106.11420, 2021.

[3] Ke Sun, Yingnan Zhao, Shangling Jui, and Linglong Kong. Exploring the training robustness of distributional reinforcement learning against noisy state observations. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 36–51. Springer, 2023.

We thank Reviewer ad2c for providing the positive feedback and recognizing the importance of our work. Please see our response to your questions below.

1. When we train different baseline algorithms (including BCQ, CQL, TD3+BC, Diffusion QL, and RORL), we use the same dataset size and batch size. By utilizing the datasets in the D4RL benchmark, we ensure that the dataset size remains fixed, thereby ensuring fairness in the comparisons made. Furthermore, we adhere to the suggested hyperparameter settings for all baseline algorithms, including setting the batch size to 256.

2. We agree that the data convergence issue of a RL algorithm significantly affects its stability and performance. In the D4RL benchmark, datasets labelled with "medium-replay" in the Mujoco domain have only 1/5 of the data compared to other datasets. In addition, a significant portion of the state-action visitations are from suboptimal strategies, and many are from random policy. In our experiments, it is found that DMBP still performs well under such circumstances (cf. Tables 1 and 2). The baseline algorithm VI-LCB recommended by the reviewer has strong theoretical guarantees. However, due to the time constraints, we are currently unable to present the simulation results of DMBP assisting VI-LCB in a scarce data setting with perturbed state observations. We will validate this in our future work.

We thank Reviewer YkkL for the compliments on the novelty of our research and valuable suggestions. We address the concerns in your review point by point below.

Response to your Questions

1. We agree that imitation learning is a class of highly effective and well-studied offline RL algorithms. Adding an imitation learning algorithm to the baseline algorithms would be valuable to demonstrate the generality of DMBP. However, given the space constraints and limited time, we struggled to do it for now. It should be noted that a class of value-based offline RL algorithms, which utilize policy regularization methods, achieve a similar effect as imitation learning, i.e., selectively cloning the policy in datasets. The baseline algorithms adopted in our experiments, like BCQ, TD3+BC, and Diffusion QL, belong to this category. Experimental results demonstrate that DMBP performs well for baseline algorithms in this category. In the revised manuscript, we have added a new section (Appendix D.6) to discuss the observed experimental results, where we analyze the performance of DMBP in comparison to policy regularization category and conservative Q estimation category offline RL algorithms. Below we present partial experimental results on the suggested algorithm IQL (Implicit Q Learning), which is trained on expert dataset in Mujoco domain.

2. There is no need to individually select the hyperparameters related to the variance schedule (i.e., $a, b, c$), and one can use the same set of predetermined values (as listed in Table 3) for all environments and datasets. We redesigned our variance schedule with the specific aim of effectively handling information with small to medium scale noises. In other words, we intentionally set $\sqrt{\bar{\alpha}\_i}$ to be large and $\sqrt{1 - \bar{\alpha}\_{i}}$ to be small for smaller diffusion timesteps $i$, which ensures that ${s}\_{t}^i=\sqrt{\bar{\alpha}\_i} {s}\_{t} + \sqrt{1-\bar{\alpha}\_i} {\epsilon}$ maintains significant raw state information.

3. In classical diffusion models, one typically assumes that the noise follows a Gaussian distribution and uses batch-generated Gaussian distributed noise for training. However, during the testing phase, the noised states are generated individually. In other words, for a single instance of $\tilde{{s}}_t = {s}_t + {\epsilon}_t$, we cannot determine the exact distribution of ${\epsilon}_t$ based on a single sample. For DMBP, during the single-step denoising process in testing, when the noise scale is given, it is trained to reconstruct ${s}_t$ from the observed $\tilde{{s}}_t$ for all possible values of the noise ${\epsilon}_t$ within a given range, which means that DMBP aims to learn a mapping that can effectively denoise the observed state within a certain range, regardless of the specific distribution of the noise. This may explain why DMBP also works well for adversarial attacks.

Response to the listed Weaknesses

1. We do agree with your point that traditional methods (e.g., the Kalman Filter for discrete control) have been proposed to address the issue of observation perturbations for discrete scenarios. However, the Kalman filter requires exact knowledge of the governing equation of the controlled system and the related parameters. In offline RL settings, on the other hand, we have no prior knowledge on the system's governing equations or parameters (otherwise, the agent can be trained online with a simulator). The only information available is the offline dataset. Modeling the environment for high action/state space problems with limited data can be prohibitively challenging. As discussed in the Introduction, modeling environment (called model-based) methods are mostly used for data augmentation in offline RL. To our knowledge, there are no relevant references on offline RL methods using the Kalman Filter for continuous control problems.

2. Thanks for pointing this out! We have fixed the issue and supplemented the data in all tables with standard deviations.