Towards Robust Offline Reinforcement Learning under Diverse Data Corruption

We thank the reviewer for the constructive comments, and we provide clarification to your concerns as follows. We would appreciate it if you have any further questions or comments.

Q1: more challenging, environments have been left out of the experimental analysis, especially in terms of the potential impact of perturbations ... I'm referring to the following environments: locomotion-v2, antmaze-xx, kitchen, adroit.

A1: Thank you for your suggestion. Our initial focus was to thoroughly study data corruption in commonly used MuJoCo environments. However, as the reviewer pointed out, we lacked experiments on more challenging tasks. To address this concern, we conducted experiments on the AntMaze benchmark during the rebuttal period. The results presented in Table 11 in Appendix I indicate that AntMaze poses a greater challenge compared to the MuJoCo benchmark. Most offline RL algorithms struggle to learn from corrupted datasets in these environments, primarily due to the high dimensionality of states and the issue of sparse rewards. In contrast, our proposed algorithm RIQL demonstrates superior effectiveness in learning policies from corrupted data, surpassing other algorithms significantly. These results further strengthen the findings in our paper and highlight the importance of RIQL. For a quick overview, we also provide the average results across four AntMaze environments below.

Q2: the theoretical analysis and the experimental evidence detailed in the section appear contradictory ... empirical results indicate that IQL still remains susceptible to dynamics attacks. This contradiction is not further examined in the manuscript. On the contrary, what is proposed is to address IQL's issue with dynamic attacks.

A2: Thank you for the valuable question. In fact, our theoretical analysis not only aligns with our empirical findings but also provides an explanation for why IQL remains susceptible to dynamics corruption. Specifically, the corruption error term in our theory scales as $\sqrt{\zeta/N}$ (ignoring other dependencies on $R_{\max}, \gamma$, and $M$), where $\zeta = \sum_{i=1}^N (\zeta_i + \log \zeta_i')$ is the cumulative corruption level. This suggests that IQL works well for data corruption scenarios when $\zeta \ll N$. In the dynamics corruption scenario, the heavy-tailed issue can cause $\\{\zeta_i\\}\_{i=1}^N$ to become very large ($\\|\cdot \\|_{\infty}$ over a heavy-tailed distribution can be very large, see definition of $\zeta_i$ in Assumption 1), thereby making the upper bound in Theorem 3 very loose. This can explain IQL's susceptibility under dynamics corruption. Furthermore, we would like to highlight that addressing the heavy-tailed issue via Huber regression considerably enhances IQL's performance. This, to a certain extent, confirms that the heavy-tailed issue is a crucial factor leading to IQL's poor performance, as the Huber loss serves as an effective method for tackling heavy-tailed regression problems (Lemma 5, Section 5.2). We have added this discussion in Section 5.2, as highlighted in blue.

Q3: Previous works on certification protocols for offline RL against attacks has not been considered for the novel RIQL model proposed. This is an important weaknesses that should be addressed in the rebuttal.

A3: Thank you for the suggestion. Upon thoroughly examining related works, we discovered that COPA [1] is the only study focusing on the certified robustness of offline RL with data corruption. However, COPA is not directly applicable to our setting due to several differences. Specifically, COPA corrupts trajectories, while we corrupt independent transitions. Additionally, COPA's certification protocol necessitates a discrete action space, whereas our investigated environments are continuous. As proposing a new certification protocol is beyond the scope of our paper, we have not conducted certification. Meanwhile, Theorem 3 in our paper appears to offer some form of "certification" for the policy learned by IQL since it guarantees that IQL achieves reasonable performance when the offline dataset is corrupted.

Q4: Minor comments: (Wu, 2022) Update the reference as it is not anymore an ArXiv manuscript but an ICLR22 paper.

A4: Thank you for bringing this mistake to our attention. We have revised this in the revision.

References

[1] Wu F, et al. COPA: Certifying Robust Policies for Offline Reinforcement Learning against Poisoning Attacks. ICLR 2022.

Q9: Does observation normalization also improve the performance of other algorithms?

A9: Thank you for your question. We implemented observation normalization for both EDAC and CQL on the Walker2d and Hopper tasks, incorporating random data corruption across four elements. The average results, as displayed below, indicate a modest overall improvement for these algorithms.

Q10: The sentence "This is likely because normalization can help to ensure that the algorithm's performance is not unduly influenced by larger-scale features." is difficult to understand.

A10: Thank you for pointing out this. To make it clear, we polish the sentence to "Normalization is likely to help ensure that the algorithm's performance is not unduly affected by features with large-scale values.".

References

[1] Sasaki F, Yamashina R. Behavioral cloning from noisy demonstrations. ICLR 2020.

[2] Liu L, et al. Robust imitation learning from corrupted demonstrations[J]. ArXiv 2022.

[3] Jin Y, et al. Is pessimism provably efficient for offline rl? ICML 2021.

[4] Zhang X, et al. Corruption-robust offline reinforcement learning. AISTATS 2022.

[5] Rashidinejad P. et al. Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism.

[6] Xie T. et al. Bellman-consistent Pessimism for Offline Reinforcement Learning

[7] Uehara M and Sun W. Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage.

[8] Kumar A, et al. Conservative q-learning for offline reinforcement learning. NeurIPS 2020.

[9] An G, et al. Uncertainty-based offline reinforcement learning with diversified q-ensemble. NeurIPS 2021.

[10] Nair A, et al. Awac: Accelerating online reinforcement learning with offline datasets. arXiv 2020.

Q5: The adversarial corruption setting is not really "adversarial" towards algorithms other than EDAC. ... the adversarial noise should be computed with respect to $Q_{\alpha}$ learned by RIQL.

A5: The adversarial corruption is not specific for EDAC, as we utilize pretrained Q functions on the clean dataset rather than the Q functions obtained during EDAC training on the corrupted dataset. Our motivation for the adversarial corruption is to perform projected gradient descent on an oracle Q function, creating a more challenging setting than random corruption. In our preliminary experiments shown below, we assessed IQL under adversarial attacks using Q functions trained by CQL, MSG, EDAC, and IQL itself. We observed that EDAC's Q functions yield the most effective attack, while IQL's exhibit the weakest attack performance. We speculate that this is due to value-based methods learning better value functions for attacking purposes. Consequently, we opted to use pretrained EDAC's Q functions to perform the attack.

Q6: The paper does not contain experiments for sequence-modeling-based algorithms such as Decision Transformer (Chen et al., 2021) or Diffuser (Janner et al., 2022).

A6: Thank your for the suggestion. To solve the reviewer's concern, we (1) add an additional discussion in Appendix A regarding algorithms that utilize transformers and diffusion models, and (2) include the results of Decision Transformer under both random and adversarial corruption (Tables 9 and Table 10 in Appendix G). Despite DT leveraging trajectory history information and taking 3.1 times the epoch time compared to RIQL, it still underperforms RIQL by a large margin. For a quick overview, we also provide the average results across three environments (Halfcheetah, Walker2d, and Hopper) below.

Q7: Minor comments

A7: Thank you for pointing out these typos. We have made revisions in our updated manuscript.

Q8: The main theorem holds for any algorithm that uses AWR to learn the optimal policy. Is it possible to add experimental results of other algorithms based on AWR?

A8: Yes, we add experimental results of AWAC [10], which utilizes a weighted imitation learning framework, in Figure 1 to provide additional validation for our theory. As shown in Figure 1, AWAC exhibits notably robust performance under observation and action corruption, despite not quite reaching the level of IQL. These results not only validate the effectiveness of weighted imitation learning in enhancing robustness but also indicate that IQL's expectile regression and detached value training contribute to further improvements, which is also evidenced by the ablation study in Appendix E.1.

We thank the reviewer for the constructive comments, and we provide clarification to your concerns as follows. We would appreciate it if you have any further questions or comments.

Q1: The paper provides no plausible scenarios where a malicious attack on the offline dataset would occur.

A1: To investigate a more challenging setting than the random corruption, we introduce the adversarial corruption, which modifies the values of state, action, reward, and next-state in an adversarial manner. This setting is particularly relevant in scenarios where malicious attackers exist. For example, some annotators might intentionally provide wrong responses for RLHF data collection or assign higher rewards for harmful responses, potentially rendering the LLM harmful for human use when learning from these annotated data. We include the example in Section 1.

Q2: The definition of $\pi_{D}(a|s)$ is unclear.

A2: Thank you for your question. The policy distribution of the corrupted offline dataset is denoted by $\pi_{\mathcal{D}}$, and its formal definition in a discrete form is given by:

$$\pi_{\mathcal{D}}(a|s) = \frac{\sum_{i = 1}^N\mathbf{1}\\{s_i = s, a_i = a\\}}{\max\\{1, \sum_{i = 1}^N\mathbf{1}\\{s_i = s\\}\\}}.$$

Here, $\mathbf{1}$ is an indicator function, and $N$ is the number of samples in dataset $\mathcal{D}$. Correspondingly, $\pi_{\mu}(a|s)$ is the behavior policy that induces the clean data. We have clarified this in the revision.

Q3: The paper assumes that IQL can learn the optimal value function $V^{*}$ from the corrupted dataset without justification.

A3: As IQL's policy and value training can be separated, the analysis of obtaining the optimal value function $V^*$ and weighted imitation policy learning can be considered as two disjoint parts, with the latter being our primary focus. In fact, we have also demonstrated that Huber regression can effectively find the optimal value function even in the presence of the heavy-tailed issue (the original IQL paper proves this in the uncorrupted setting without the heavy-tailed issue). Please refer to the discussion in Appendix C.2 for details. We have revised the main paper to provide justification for this.

Q4: (1) If environmental noise exists, the entire dataset would be corrupted, not just a tiny portion. (2) Adding random noise just to $s$ or $s'$ does not seem to make much sense.

A4: Thank you for the valuable questions. Regarding the first question, our rationale is based on the fact that in many real-world scenarios, engineers often encounter a combination of high-quality data and poor-quality data. This mixed data setting is common in robust imitation learning [1][2] and aligns with the $\epsilon$-Contamination model in theoretical analysis [3]. In Section 6.2, we have investigated varying corruption rates from 0 to 50$\\%$, which is not a tiny portion. In cases where the entire dataset can be corrupted, it can be considered a special case in our setting by adjusting the corruption rate to 1.0.

Regarding the second question, we remark that the use of full trajectories as the offline dataset is the typical choice in the context of finite-horizon MDPs (e.g., [3]). However, under the framework of discounted MDPs, storing offline RL datasets as independent 4-tuples is a commonly used approach in both theoretical literature [4][5][6][7] and empirical works [8][9]. Particularly in TD learning, states $s$ and next-states $s'$ play distinct roles: the former defines the input distribution, while the latter determines the future outcome. Consequently, they can be independently corrupted. In our experiments, we also observed different effects when corrupting $s$ and $s'$: the Huber loss can alleviate dynamics corruption, but it cannot mitigate observation corruption.

Thank you for your reply. I want to make some clarifications regarding Q4-2, though.

1. We can preserve the sequential information without allocating memory based on the longest trajectory by saving the data as a sequence of (observation, action, reward, truncated, terminated) 5-tuple. This is exactly how D4RL stores data.

2. For MDPs, the 4-tuple view is sufficient, but for POMDPs, we need to preserve the sequential information. Most real-world problems are not Markovian, so the sequential information has to be preserved.

To sum up, even if an algorithm adopts the 4-tuple approach, it is better to store the data as a sequence of (observation, action, reward, truncated, terminated) 5-tuples and then convert it to a collection of $(s, a, r, s')$ 4-tuples in memory during the learning process. As attacks on the memory values are extremely difficult, I think assuming corruption would happen independently on s and s' is unrealistic.

We thank the reviewer for the constructive comments, and we provide clarification to your concerns below. We would appreciate it if you have any further questions or comments.

Q1: other new baselines using a transformer or a diffusion model. It is helpful to involve the discussion about such algorithms at least.

A1: Thank you for the suggestion. We include a discussion in Appendix A regarding algorithms that utilize transformers and diffusion models. Additionally, we compare our method to Decision Transformer (DT) in both the random and adversarial corruption benchmarks, as shown in Tables 9 and Table 10 (Appendix G). Despite DT leveraging trajectory history information and taking 3.1 times the epoch time compared to RIQL, it still underperforms RIQL by a large margin. For a quick overview, we also provide the average scores over three environments (Halfcheetah, Walker2d, and Hopper) below.

Q2: add some baselines that also use ensemble Q, while not appearing in the experiments if I didn't miss something.

A2: In our initial submission, we have included SOTA ensemble-based baselines, such as EDAC [1] and MSG [2]. While these baselines demonstrate exceptional performance on clean data, we note that they are considerably more vulnerable to data corruption, with a performance drop of approximately $60\% \sim 70\%$, as presented in Table 1 and Table 2 (Section 6.1).

Q3: no comparisons between the proposed algorithm RIQL to other existing robust RL algorithms

A3: To the best of our knowledge, prior to our submission, there were no empirical robust offline RL baselines designed for the data corruption setting. Previous research primarily focused on theoretical guarantees [3], certification protocols [4], and evaluations [5] for offline RL with data corruption. However, we find a recent study [6] that introduced a solution called UWMSG for addressing reward and dynamics corruption, and we compare it with our proposed method.

The comparative results can be found in Table 9 and Table 10 (Appendix G). Notably, RIQL demonstrates significantly improved performance compared to UWMSG. We believe that our work will serve as a strong baseline for future research in this setting. For a quick overview, we have also provided the average scores across three environments below.

Q4: Is there any study on ablation study for the proposed RIQL that is similar to Figure 1. It will be helpful to see how can RIQL handle different types of data corruption.

A4: Thank you for the suggestion. In Appendix H, we include an additional ablation study under separate observation, action, reward, and dynamics corruption. These results are helpful for understanding how each component of RIQL contributes to performance under various data corruption scenarios. For example, from the results, we conclude that the Huber loss is more beneficial for reward and dynamics corruption, but less effective for observation and action corruption. Overall, all three components play an important role in enhancing the performance of RIQL.

References

[1] An G, et al. Uncertainty-based offline reinforcement learning with diversified q-ensemble. NeurIPS, 2021.

[2] Ghasemipour K, et al. Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters. NeurIPS, 2022.

[3] Zhang X, et al. Corruption-robust offline reinforcement learning. AISTATS, 2022.

[4] Wu F, et al. COPA: Certifying Robust Policies for Offline Reinforcement Learning against Poisoning Attacks. ICLR 2021.

[5] Li A, et al. Survival Instinct in Offline Reinforcement Learning. NeurIPS 2023.

[6] Ye C, et al. Corruption-Robust Offline Reinforcement Learning with General Function Approximation. NeurIPS 2023.

We thank the reviewer for the constructive comments, and we provide clarification to your concerns below. We would appreciate it if you have any further questions or comments.

Q1: Lack the theoretical evidence on heavy-tailed Q-target issue

A1: The identification of the heavy-tailed issue primarily stems from empirical observations, including visualization and the calculation of Kurtosis values. It is challenging to provide a rigorous mathematical explanation for the cause of the heavy-tailed issue. However, we offer some intuitive speculation for this issue — dynamics corruption can corrupt $s'$ to less frequently visited or even artificial states $\hat s'$, whose values $V_{\psi}(\hat s')$ exhibit higher uncertainty, thereby increasing the variance of the Q-target during offline training. Furthermore, we want to underscore that many existing works [1][2] in this field also identify heavy-tailed problems empirically and adopt assumptions similar to Assumption 4 in our paper. This assumption is mild in that it only requires the existence of $(1+\nu)$-th moment, and our theoretical analysis (Lemma 5, Section 5.2) demonstrates why Huber regression can effectively address the heavy-tailed issue.

Q2: Heavy work on parameter finetuning

A2: Due to varying inherent properties of different attacks and environments, optimal hyperparameters differ. However, after conducting a parameter search, we find that in most settings, $N=5, \alpha\in\{0.1, 0.25\}, \delta\in\{0.1,1.0\}$ yields the best or comparable performance. This hyperparameter space is relatively small. Additionally, we provide a hyperparameter study in Appendix E.8 to guide hyperparameter tuning. For instance, it is recommended to use larger $\delta$ (e.g., $1.0$) to address dynamics and reward corruption, as opposed to smaller values (e.g., $0.1$) for observation and action corruption.

Q3: making it more clear about the mechanism that generates the corrupted data in each case, and how to ensure that the generated noisy data represent typical noisy data in practice?

A3: In Appendix D.1, we present a comprehensive explanation of the generation of corrupted data, where we consider two major forms of corruption: random corruption and adversarial corruption. Random corruption is similar to random noise encountered in measurement due to the inherent variance of the device or external factors in the environment. To simulate this scenario, we introduce Gaussian noise for states, actions, and next-states. To investigate a more challenging setting, we introduce adversarial corruption, which modifies the values of state, action, reward, and next-state in an adversarial manner. This setting is particularly relevant in scenarios where malicious attackers exist. For example, some annotators might intentionally provide wrong responses for RLHF or assign higher rewards for harmful responses, potentially making the LLM harmful for human use when learning from the annotated data. The two types of corruption also align with prior related works in robust imitation learning [3] and robust RL [4].

Q4: different levels of noise would lead to different robustness outcomes - how do you ensure that the empirical evidence is conclusive?

A4: We agree that different levels of noise can yield different robustness outcomes. In our setting, the noise level is determined by two factors: the corruption scale (the magnitude of each added noise) and the corruption rate (the proportion of data that is corrupted). In Section 6.2, we assess the performance of different algorithms across varying corruption rates (0$\%$ $\sim$ 50$\%$) and demonstrate that RIQL exhibits greater robustness compared to other algorithms at different corruption rates. Additionally, we present results for random and adversarial corruption at scale 1 in Section 6.1 and scale 2 in Appendix E.6. Our findings indicate that RIQL consistently outperforms other algorithms. Considering that many baselines already attain scores lower than 10 under corruption scale 2, which is a notably low score, and taking into account the fact that noise with too large scale can be more easily detected in real-world scenarios, we have not continued to increase the corruption scale.

References

[1] Garg S, et al. On proximal policy optimization’s heavy-tailed gradients. ICML, 2021.

[2] Zhuang V, Sui Y. No-regret reinforcement learning with heavy-tailed rewards. AISTATS, 2021.

[3] Sasaki F, Yamashina R. Behavioral cloning from noisy demonstrations. ICLR, 2020.

[4] Zhang H, et al. Robust deep reinforcement learning against adversarial perturbations on state observations. NeurIPS, 2020.